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ABSTRACT 


The nature of warfare has changed dramatically during the last decade. Western 
armies are increasingly required to conduct complex operations in urban terrain against 
asymmetric threats. These opponents use cities and their inhabitants for cover and 
concealment. In such situations, modern equipped armies often cannot fully utilize many 
of their most powerful weapons. To overcome this situation, modern communication 
systems are being acquired and deployed to provide real-time reconnaissance; thereby, 
attempting to neutralize the threat through enhanced situational awareness. This research 
addresses the potential impacts of communication from airborne sensors on assisting a 
convoy in finding its way through a hostile city quarter (based on Mazar-E-Sharif, 
Afghanistan) in which militia forces try to interdict them via street blockades and 
ambushes. The implementation is done in the agent-based simulation Map Aware Non- 
Uniform Automata (MANA). The results show that the current MANA version is not 
sufficiently capable to handle routing problems in urban terrain. Specifically, the 
movement algorithm is “locally greedy” and not flexible enough to project into the 
future—as real human decision makers do. Many workarounds were developed to 
mitigate this limitation. The analysis shows that the number of blockades is the single 
most important factor in determining the convoy’s success. Of the communication 
factors, network latency has the most impact. For the convoy to effectively use the 


information, it needs to get from the sensor to the convoy in 11 seconds. 
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EXECUTIVE SUMMARY 


The nature of warfare has changed dramatically during the last decade. Western 
armies are increasingly required to conduct complex operations in urban terrain against 
asymmetric threats. These opponents use cities and their inhabitants for cover and 
concealment. In such situations, modern equipped armies often cannot fully utilize many 
of their most powerful weapons. To overcome this situation, modern communication 
systems are being acquired and deployed to provide real-time reconnaissance, thereby 
attempting to neutralize the threat through enhanced situational awareness. This research 
addresses the potential impacts of communication from airborne sensors on assisting a 
convoy in finding its way through a hostile city quarter in which militia forces try and 


interdict them via street blockades and ambushes. 


The vignette chosen to explore the communication aspects in urban combat is 
inspired by the “Black Hawk Down” incident in Mogadishu, Somalia. As depicted in the 
movie, a convoy has to make its way through a hostile city neighborhood in which militia 
forces build blockades and try to ambush the convoy. Unfortunately for the convoy, in 
Mogadishu, the delay in getting information from their airborne reconnaissance platforms 
(helicopters) made the information almost useless. In our situation, the convoy is trying 
to navigate through terrain based on Mazar-E-Sharif, Afghanistan. The airborne sensor 
grid supporting the convoy consists of hovering Unmanned Aerial Vehicles (UAVs). 


The UAVs provide near real-time information about blockades. 


The scenario is implemented as an agent-based simulation using the software 
package Map Aware Non-Uniform Automata (MANA), see Figure 1. However, 
transferring a classic network routing problem into an agent-based model is not easily 
done. The software’s limitations make it necessary to use many workarounds. In 
particular, the convoy’s movement decisions—which MANA limits to greedy local 
choices—have to be made more robust. The partial screen shot (Figure 2) shows the 


main ideas in our approach to making the convoy behave reasonably. 


XVil 
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Figure 1. The original map of Mazar-E-Sharif and the MANA terrain map. 
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Figure 2. Helper agents used to assist the convoy in utilizing information. (Best 
viewed in color) 

The convoy can rely on a grid of friends, comparable to the street map. These 
friends draw the convoy into usable streets (green) or warn it and repel it when the street 
is blocked (yellow). Having additional information from the sensor grid, the convoy is 
informed by these ‘logic helpers’ on which roads are clear. An important question is, 


how do these workarounds influence the model’s quality and usability? 


In order to settle the analysis on a broader base, the factors are divided into 
uncontrollable and controllable sets. The factors the convoy cannot control include the 
number and placement of blockades. The controllable factors are those associated with 


XViil 


its sensors and communications network. These include latency, reliability, capacity, 
sensor range, and update rate. Nearly Orthogonal Latin Hypercubes are used to 


efficiently conduct the experiments. 


Two experimental sets are conducted. The first experiments help assess the 
model’s quality by comparing the convoy’s ability to successfully reach its goal under 25 
blockade combinations, both with communications and without communications. That 
is, without communications the convoy must rely only on its own visual sensor; with 
communications the convoy can also take advantage of the information given to it by a 
grid of hovering UAVs. The measure used to quantify the effectiveness of the convoy in 
traveling to its objective is the number of hits the convoy suffers. The main result from 


the quality assessing experiment is shown in Figure 3. 
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Figure 3. Proportion of times the convoy with communication does better, equal, or 
equal & better than the convoy without communications for 25 separate 
combinations of blockades. The number of blockades increases with the 
x-axis. (Best viewed in color) 


The experiment was run to compare the convoy’s behavior with and without 


communications. The graphical analysis reveals that for a large number of blockades the 


X1x 


convoy with communications does better only about 60 percent of the time—a 
disappointingly low number. On the other hand, the statistical analysis shows that 
communications has an improving impact by reducing the mean number of hits the 


convoy suffers for all 25 blockade configurations. 


The second experiment, the main focus of this thesis, examines the 
communications factors. The 25 combinations of blockades are tested against 65 
communication settings. This experiment contains 1625 input combinations, with 100 
replications each. The analysis of the output data from these 162,500 computational 
experiments shows that the number of blockades has the most impact on the convoy’s 
success. Unsurprisingly, latency within a communication network is the most important 
communications factor, followed by the pause (delay time) between the information 
updates given by the sensor grid. That is, even good information in such a situation has 
to be timely, or it is nearly worthless. The exclusion of other factors, such as reliability 
and capacity, must be taken in the context of our static scenario—i.e., there are no 


moving enemies. Figure 4 shows part of a partition tree on the communications factors. 
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Figure 4. A portion of the regression tree with the partitions made only on 
communications factors. The times are in seconds. We see the 
importance of the hovering sensors getting information quickly to the 
convoy, and its effect on the mean number of hits the convoy suffers. 
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Other lessons learned and key takeaways from implementing this scenario in 


MANA include: 


The current simulation software complicates the realistic modeling of 
movements in urban terrain, and will continue to do so as long as a routing 
problem is involved. The necessity of an underlying network structure for 
movement decisions became obvious to the author. 


The communication links provided by the software package do not reflect 
the current technology standards. In order to model realistic digital 
communication networks, workarounds had to be_ used. These 
workarounds reduced the simplicity and the idea of agent-based models. 
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I. INTRODUCTION 


A. BACKGROUND 


A military unprepared for urban operations across a broad spectrum is 
unprepared for tomorrow. [Peters 1996] 


Since the fall of the iron curtain in 1990 and the end of the Cold War, warfare has 
changed drastically. The old doctrines were modeled on the expectation that nearly all 
decisive battles would take place in open terrain. Both combatants in the Cold War— 
NATO and those bound by the Warsaw Pact—clearly desired to avoid combat in urban 
environments. Behind this intention to steer conflict away from cities was the possibility 


of an incalculable number of casualties in urban zones. 


On the other hand, “control of cities always has been vital to military success, 
practically and symbolically.”! Today, Western armies face more and more asymmetric 
threats in which the opponent attempts to draw the forces into an urban environment. 
These opponents are familiar with the area and many of the high-technology weapons of 
the West are largely ineffective. Furthermore, opponents use civilians in the cities as 
shields and restrict the engagement of weapon systems which are designed for open space 
combat. The current situation in Baghdad illustrates the problems that Western forces 
encounter in this new urban combat. American political and military leaders are clearly 
focused on securing the city. In fact, success in this major city will likely determine the 
success of the coalition forces in all of Iraq. In recent months, however, terrorist 


organizations have attempted to destabilize Baghdad. 


The likelihood of troop engagement in cities is increasing. One factor in this new 
development, as mentioned before, is the opponent’s strategy to use the advantage of 
familiar urban terrain and the cover of non-combatants in the streets. On the other hand, a 
modern equipped Western army lacks the opportunity to unfold its total power. As a 
result, this army experiences a reduced effectiveness of high-tech weapon systems and 


limited communications. Another aspect of increases in urban combat is the nature of 


1 Ralph Peters, “Our Soldiers, Their Cities,” Parameters, Spring 1996, pp. 43-50. 
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cities themselves. Demographic data shows that urbanization is an ongoing process. 
Cities act like a center of gravity, attracting people to move there. Currently, half of the 
world’s population lives in and around cities, and a prognosis for the next century shows 


that about 70 percent of people will make their homes in an urbanized terrain.” 


Even in low-intensity peacemaking and peacekeeping missions like NATO led 
KFOR in Kosovo, forces must be deployed in urban environments. “Operating in a 
potentially hostile city is every soldier’s nightmare.”3 Behind each corner, out of every 
window, the enemy can attack the patrolling soldiers. These kinds of operations are 
challenging for a modern army. In a training mission conducted in Oakland, California, 
U.S Marines attacked an Old Navy Hospital and suffered high losses (70%) during the 
first hours of combat.4 At the time, most Western armies were ill-prepared for urban 


operations. 


In order to overcome this miserable situation, military organizations have recently 
made several changes in the preparation and training of their soldiers. Physical and 
psychological training are necessary in order that the soldier may avoid distraction caused 
by a totally destroyed environment and the human misery caused by war. The militaries 
have also improved individual equipment. Modern modular gear contains night vision 


goggles and a rifle with laser ranger, compass and head-ups display. 


The key factor to increase survivability in urban combat, however, is and will be 
information sharing and collaboration. In a network-centric operation, each soldier 
benefits from precise information. Consequently, the situational awareness, speed of 


command and mission effectiveness improves dramatically.> 





2 Jean Kumagai, “Fighting in Streets,” IEEE Spectrum, February 2001. 


3 Gerald Yonas and Timothy Moy, “Emerging Technologies and Military Operations in Urban 
Terrain,” in Michael C. Desch “Soldiers In Cities; Military Operations On Urban Terrain,” 2001. 


4 Jean Kumagai, “Fighting in Streets,” IEEE Spectrum, February 2001. 


5 Definition of Network-centric operations by MORS online, www.mors.org, October 2006. 
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“Real-time and detailed information is critical to the success of ground combat 
forces.” © Over-the-hill-reconnaissance done by a networked, unmanned aerial sensor 
system provides the commanding officer pertinent information. Thus, troops can cover 


more area and gain situational awareness to aid them in the hostile environment. 


Successful reconnaissance systems will therefore meet the following 


requirements:? 


e Detect, identify, and track enemy targets, 
e Be equipped with robust, secure and stealthy sensor arrays and 
e Contain jam resistant links for sensor fusion and command and control. 


Furthermore, in order to reach the commanding officer or fighting soldier in real-time, 
data processing from the sensor to command and control must be automated. Challenges 
such as the amount of collected data and making comparisons with databases to 
recognize and classify an enemy correctly are expected to be solvable with technology. 
By automating these data intensive procedures, human intervention would normally be 


excluded from this process and thus would reduce latency. 


B. GERMANY’S C4ISR APPROACH 

At this time, the German Armed Forces does not possess fully networked 
capabilities. They have recognized, however, that C4ISR (command, control, 
communications, computers, intelligence, surveillance and reconnaissance) 1s the heart of 


modern warfare.’ 


Efforts to achieve a network-centric military are often slowed by budget concerns. 
The lion’s share of the German military procurement budget is spent on a few major 
acquisitions (e.g., Eurofighter, air-transporter AIRBUS 400M, Tiger and NH- 
helicopters). The German military must purchase all of these investments in fulfillment 


of long-binding contracts with German and European companies. 





6 Gene Klager, “Networked Sensors for the Combat Forces,” in “Unmanned/Unattended Sensors and 
Sensor Networks,” Edward M. Carapezza (Editor), November 2004, pp. 204-214. 


7 Jay Nemeroff, Luis Garcia, Dan Hampel, Stef DiPierro, “Application of Sensor Network 
Communications,” IEEE 2001. 


8 Gordon Adams, Guy ben-Ari, John Logsdon, Ray Williamson, “Bridging the Gap,” October 2004. 
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Another obstacle to progress in networking capability is the lack of inter- 
operability among existing C3 equipment. For example, HEROS (Army command and 
control system for digitally-supported command operations in staffs) is introduced to 
corps, divisions and brigades. This system has no interface to the lower echelon system 


FAUST (command and control provision) for regiments and battalions. 


The FAUST system is mounted on most of the vehicles and tanks operating in 
Afghanistan and Kosovo and delivers a variety of support for the forces. Its updated map 
mirrors the friendly forces’ positions and the reconnaissance results about the enemy that 
have been entered manually. Other tools are also helpful, such as emergency call, free 


text messages and automated updates and reports concerning the position of the vehicle. 


The German Army has one successful UAV at its disposal. The drone CL-289 is 
able to take black/white or infrared pictures and deliver real-time video data. These 
results must be analyzed by operators and entered into the C3 software packages. Other 


UAV programs have been abandoned during recent years due to budget limitations. 


The German Army will attempt to overcome their technological handicap by 
achieving interoperability between the C3 systems and with other branches and allies. 
The automated data entries are absolutely necessary to gain a real-time information flow 


in an ongoing operation. 


New UAV programs have been launched to improve over-the-hill reconnaissance 
capabilities with the forces. One such program has resulted in the FANCOPTER (shown 
in Figure 5), developed by EMT. The FANCOPTER is a small hovering UAV specially 
designed for missions in urban terrain or indoors. With a rotor diameter of half a meter, 
the UAV stabilizes in a position by using acceleration sensors which measure the wind 
speed. The FANCOPTER may be equipped with different camera types, including 
digital, video, or IR camera. At this stage of development, its radius of action is about 


500 meters, and the maximum mission length is about 20 minutes. 





Figure 5. _FANCOPTER developed by EMT Penzberg. (From: www.emt- 
penzberg.de (August 2006)). 


A fully functional C4ISR system is essential to the German Army. Since the 
reunification in October 1990, the German Armed Forces have begun to support United 
Nations missions. As a NATO member, Germany also took part in the Kosovo invasion 


after the air strikes in 1999. 


For the last four years, Germany has contributed a large contingent to the 
stabilization forces in Afghanistan (ISAF). The latest news shows that the war on 
terrorism has not yet been won in this region. The threat to German soldiers in this 


mission abroad is constantly increasing. 


With the transfer of the main contingent from Kabul to Mazar-E-Sharif 
completed, Germany has taken over the responsibility for the Northern Province. A map 


of Mazar-E-Sharif is used for the vignette presented later in this thesis. 


The vignette in this thesis is modeled in MANA. This agent-based simulation is 
well-accepted in the simulation community. The scenario is also implemented in IT- 
SimBw. Besides the development and procurement of modern communication 
technology, “The German Armed Forces Federal Office for Information Management 
and Information Technology,” in cooperation with Fraunhoffer Institute, started to build 
IT-SimBw as a new agent-based simulation tool that will examine communication 
aspects of combat. The first demonstrations of this new software package were done 
during the Project Albert workshops in Honolulu in February 2006 and Boppard, 
Germany in June 2006. Modeling insights experienced during this research have proved 


helpful to the developers of IT-SimBw. Similarly, some limitations of the widely used 


“simulation software MANA have become apparent during the course of research 
conducted for this thesis. It is not within the scope of this thesis, however, to make a 


comparison between these simulation packages. 


C. SCOPE 

The thesis research operates on the basic assumption that the employed forces 
have a modern C4ISR system available for a specific mission. The listed questions of 
interest are addressed to explore the communication aspects of fighting in urban terrain: 


e What are the impacts of various factors (latency, reliability, data updating 
frequency, capacity and sensor ranges) for fully networked forces engaged 
in a convoy mission in an urban environment? 


e What are the requirements for agent-based simulation software to be 
effectively used to examine communications in urban terrain? 


D. FLOW OF THESIS 

This brief introduction is followed by a discussion of the simulation used, 
scenario overview, and measures of effectiveness (MOEs). The next section provides a 
detailed discussion of the design of the experiment which is applied to gain insights from 
the model. All of the parameter settings for the data farming are explained and justified. 
The subsequent section contains a statistical and analytical evaluation of the collected 
data using regression techniques and data mining tools (e.g., regression trees) in order to 
extract profitable insights. Finally, the findings are summarized and recommendations 
are given on further studies that will broaden our understanding of aspects of 


communication in combat situations. 


I. MODEL DEVELOPMENT 


This chapter deals with the model development. After explaining the scenario 
background and deriving the vignette, the simulation software MANA is briefly 
examined, including the software’s limitations and the associated consequences for the 
model. The next section introduces the model terrain and time with their relation to real 
world measures. This is followed by an outline of the agents and their main behaviors. 
The scenario script visualizes the sequence of events during a run. The final section 


discusses the measures of effectiveness for the model. 


A. “BLACK HAWK DOWN” SCENARIO 
As discussed in the introduction to this thesis, the likelihood of combat in urban 
terrain is increasing. Initial thinking about successful support for ground forces during 


city missions began after the first unsuccessful combat in the post Cold-war era. 


Filmed in the movie ‘Black Hawk Down,’ a failed raid on a gang leader 
stronghold was conducted by US forces on October 3, 1993 in Mogadishu. US Rangers 
were attempting to capture two high-ranking lieutenants serving Farrah Aidid, the most 
dangerous warlord in Somalia. The troops, sent in by helicopter, were to be picked up by 
a convoy and brought back to the base. The two targets were arrested successfully. 
However, the Somali militia reacted quickly to the attacking US soldiers and gathered 
behind the cover of weapons in the streets. The militia then built blockades with burning 
tires and cars. For the US forces, this was a new, unexpected development in comparison 


to previous raids. 


During the campaign, two Black Hawk helicopters were shot down and the pilots 
were surrounded by militia and a crowd of aggressive civilians. The convoy was ordered 
to retrieve all soldiers from the first crash site. The situation became confused, however, 
because the command cell in a hovering helicopter above the city could not lead the 
convoy through the streets in avoidance of blockades and hostile fire. The convoy 


ignored the last order and focused instead on finding a route back to the base. 


Thousands of Somalis took to the streets and fought against the outnumbered US 
Rangers who were left in the city. The US military launched a rescue campaign 
involving Pakistani tanks and APCs. The rescuers liberated all of the soldiers—whether 
dead or alive—although one pilot was held as hostage by Farrah Aidid and released after 
several weeks in Somali captivity. On this day, 18 American soldiers died and 73 were 


injured. 


The vignette examined in this thesis research is only one part of the ‘Black Hawk 
Down’ scenario and is translated into a current German background. Several aspects of 
this scenario, which is commonly used for peace-enforcement situations, were examined 
in previous studies. In order to find insights about communication aspects in urban 
environments, this study focuses on the convoy’s attempt to make its way back to base. 


Convoy missions are an everyday business for German troops in Afghanistan. 


1. Vignette Description 

The vignette is translated to a newer world. German ground forces in Mazar-E- 
Sharif, Afghanistan attack a gang leader stronghold and arrest the suspects. The forces 
are picked up by the convoy, which then must make its way through the street channels 
and back into the secure area where the base is located. The forces are equipped with 
FAUST, which has automated data updates from a sensor grid consisting of five hovering 


UAVs (e.g., FANCOPTER). 


The well-prepared militia forces react quickly to the attack and build up blockades 
with different time delays to slow the convoy’s movement and lead it into an ambush. 


These blockades appear in three waves with random compositions. 


B. SIMULATION SOFTWARE AND ITS LIMITATIONS 

1. MANA 

The software package used for implementation is MANA Version 3.2.1. Map- 
Aware-Non-Uniform Automata (MANA) was developed by the New Zealand’s Defense 
Technology Agency. In the user’s manual,? the developers state that they developed the 


9 MANA Version 3.0 Users Manual, D. Galligan, M. Anderson, M. Lauren, July 2004. 
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new software because their existing agent-based simulation produced outputs which 
differed radically from combat model’s results like CAEn and Janus. The intention was 


not to improve a cellular automaton model, but to create a new model with new features. 


An agent-based combat model (ABM) contains entities that are controlled by 
decision-making algorithms.!° The agents react and interact with their surroundings 
(e.g., terrain, friends and enemies). MANA incorporates several new elements. For 
instance, the squad possesses two common situational awareness maps. These ‘memory 
maps’ !! store the organic information !? and inorganic information gained by 
communications. The agent can react differently to the kind of information it receives. 
The agent can also change its personality by an event-driven trigger state change. Agent- 
based models are usually time-stepped models, but as a part of the interactions with their 
surroundings, MANA agents can change their trigger states and behaviors. The duration 


in a State is variable. 


Agent-based models are intended to provide a straightforward way to create a 
scenario in order to examine a special situation. In comparison to a highly-detailed 
combat model, the run-time is relatively short. Therefore, multiple runs over a range of 
input values are feasible. An efficient design of experiment combined with statistical 
analysis provides numerous insights and sometimes surprising results. Due to the agent’s 
simple set of rules, however, it is possible that realization of ‘correct’ behavior may be 
more difficult to achieve. 

2. Limitations and Workarounds 

The developers of MANA strongly emphasize that the simulation software is not 
free of limitations. Two major limitations have become obvious: the greedy movement 
algorithm and the implementation of the communication features. These problems and 


their respective solutions are discussed in this section. MANA offers a number of 


10 MANA Version 3.0 Users Manual, D. Galligan, M. Anderson, M. Lauren, July 2004. 
11 MANA Version 3.0 Users Manual, D. Galligan, M. Anderson, M. Lauren, July 2004. 


12 Contacts detected by the agents themselves or by another squad member are stored as organic 
information on the organic situational awareness map. On the other hand, contact data that the agents 
receive by a communication link is considered inorganic information, and is stored on the inorganic 
situational awareness map. 
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possibilities to circumvent the limitations by creative modeling. Varying parameters and 
the use of different trigger states can often help to overcome the software’s weaknesses. 

a. The Greedy Movement Algorithm 

As mentioned in the previous section, the MANA movement algorithm 
operates on the agent’s desires and the terrain. In order to calculate the penalty function, 
the software takes into account only the pixels surrounding the agent. The penalty is 
updated every time step. Projections into further steps are not possible in the software 
version used for this thesis. The agent makes its decision locally according to its desires. 
This differs substantially from human decision making. In a real-life situation, the 
convoy leader gathers all available information and processes it. His time horizon is 
wider, and he bases his decision on the given situational awareness and projections into 


the future. 


In order to achieve similar agent behavior, two kinds of supporter agents 
have been created. The missing knowledge about the map is replaced by a grid of friend 
agents that send their positions constantly and reliably and act as a warning system when 
a street is blocked or there is no continuous route after using a street. The logic for 
closing streets is accomplished by ‘logic helpers,’ which use reaction chains in special 
situations. For example, when the ‘logic helpers’ at a four-way crossroad notice that 
three of the streets are blocked, then they will block the fourth street. Thus, these agents 
help the convoy avoid an ambush. 

b. The Communication Features 

The communication links implemented in MANA do not mirror the reality 
of current communication in combat. The data transmission between two squads is a 
continuous flow of information. The sender transmits its organic situational awareness 
map every time step. A message containing the enemy type and its location eventually 
reaches the recipient with high probability even if the reliability for the communication 


link is low. The high number of trials makes this possible. 


It is difficult to interpret the capacity and reliability of the communication 
factors in MANA because of the fact that, in reality, messages are sent in packages. In 


other words, the package may or may not reach the other squad. In MANA, each contact 
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is put in a single message and sent with the pre-set reliability level. A partial transfer is 
possible and can produce absurd situations. This means that the convoy can have 
knowledge about an ambush without seeing all associated blockades. Unfortunately, 
there is no perfect solution to this problem, a drawback which will be addressed in the 
results discussion of Chapter IV. 

3. Aggregation 

Aggregation is used to reduce the complexity of the model. Agent-based models 
by themselves are based on simplicity. Therefore, the convoy is modeled on the 
movements of only one vehicle. The intra-convoy behavior (movement in line and intra- 


convoy communications) is outside the focus of this research and is excluded. 


The same approach and argumentation is used for the blockades. As a military 
rule, blockades are observed, and the blockade is merged with the militiamen. That is, 


the blockade itself is armed. 


In modern communication architectures, the sensors are linked to a Command & 
Control Center (CCC) that distributes the processed data back to the deployed forces. 
Because MANA does not allow the user to disconnect or connect a communication link 
by changing the trigger state, it is nearly impossible to explicitly model the CCC. 
However, the necessity of modeling such an element is questionable. The physical 
parameters like latency, reliability and duration between updating can be viewed as 
network parameters. The latencies in the sequential network add up, while the resulting 
reliability is the product of the reliabilities along the path. The updating between sensors 
and the CCC on one side and the CCC and the ground forces should be synchronized. 
Therefore, the duration between updates is a measure for the update frequency of the total 


network. 


C. TERRAIN AND TIME 

The German Army has forces deployed in the Northern Province of Afghanistan. 
The German Headquarters is located in Mazar-E-Sharif. For the purpose of this scenario, 
one square kilometer out of the Army city map was scanned. Microsoft Paint was used to 


draw the contours and color the map. 
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In the scenario, only three terrain colors are used: black (streets), grey (buildings) 


and white (open terrain). The movement parameters are adjusted to these colors. 


The scenario terrain is not a 100 percent copy of the map. In order to gain more 
realistic convoy behavior, the streets have uneven width and the comers are softened. 
The amount of open terrain is reduced to avoid unwanted interactions. The areas around 
the cemetery and refugee camp are totally remodeled. The differences can be observed in 


Figure 6, which shows the scanned map and its ‘translation’ for the scenario. 








task 


Figure 6. The original map of Mazar-E-Sharif and the MANA terrain map. 


The resolution employed in MANA is 1000x1000 pixels. To ensure an easy 


calculation, one pixel represents one meter in the real map measure. 


The scale for one time step is designed as follows. In a peaceful environment, the 
convoy’s route stretches about 1.5 km long; it takes about 1500 time steps to reach the 
final destination. Operating on the assumption that a convoy runs through a town with an 
average speed of 30 km/h, 1500 time steps in the scenario are equal to 3 minutes (180 


seconds). One time step is thus about 0.12 seconds. 
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D. AGENTS ON THE BATTLEFIELD 
This subsection describes the main characteristics of agents within the simulation. 
Furthermore, the section emphasizes the role of workarounds. 

1. Map Grid 

As mentioned in the previous section, the convoy in the simulation does not have 
a map. It reacts only to the local terrain and the desires it possesses according to its pre- 
defined personality. Therefore, friend agents have been created to support the convoy in 
its course through the streets of the city. As dictated by the real street network, the agents 
are subdivided into three groups whose number depends on the length of the street for 
which they are responsible. They are located at the beginning and end of various streets. 
Depending on the street length, an agent has a sensor range of 30, 60 or 140 meters to 
guarantee that the agent can detect something only in its street. By default, the map grid 
is completely concealed in order to avoid interactions. The map grid has a 100 percent 
reliable and continuous communication link to the convoy. The locations of all squad 
members are transmitted. This creates a kind of street network on the convoy’s inorganic 
map. The high reliability appears as the digital map on a monitor mounted in front of the 
convoy leader. The behavior of each agent on the digital map is characterized by three 


functions. Each case is shown in Figure 7. 





Figure 7.___ The three possible states of a map grid agent. (Best viewed in color) 
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e If the street is free and usable for the convoy, these agents have the same 
allegiance as the convoy. 


e If an agent detects a blockade red Cross in its street (threat level 2), it 
changes the allegiance and becomes an unconcealed enemy with threat 
level 1. This action is identical to leaving the squad; therefore, the agent’s 
location is no longer transmitted to the convoy. 


e If an agent detects a ‘logic helper’ in its street (yellow star enemy threat 
3), it becomes an unconcealed neutral. As in the case of a red Cross, it 
leaves the squad and vanishes from the inorganic map of the convoy. 


The convoy’s reaction to the different states of map grid agents will be described in the 
section on convoy behavior. 

2. Blockades 

The scenario consists of three blockade squads (A, B and C) consisting of varying 
numbers of agents (12, 15 and 18). Blockade A becomes active when the convoy starts 
its movement. Blockade B is hidden for 215 time steps (26 seconds) and appears with its 
resulting ‘logic helpers.’ Finally, the last wave of blocked streets, blockade C, is 


activated after another 545 time steps (65 sec). 


The scenario setting is organized in the first phase of the simulation. The 
blockade-agent goes through a sequence of states, as illustrated in Figure 8. During the 
RUN START state, the agent remains totally concealed; after a fixed duration, the agent 
transitions into SPARE 1 STATE, where it becomes visible and vulnerable to attacks by 
super-agents. If the agent is killed, it disappears; otherwise, it steps into SPARE 2 
STATE. The now invulnerable blockade-agent rests in the state for a fixed amount of 
time in order to set up ‘logic helper’ correctly. When this preparation is complete, the 
agent transitions into SPARE 3 STATE, where it hibernates, totally concealed. The 
blockade remains fixed up to the point in the scenario timeline when its blockade wave is 
activated. The active phase starts with the DEFAULT STATE. The durations may vary 


depending on the blockade wave and the number of necessary logical chains. 
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RUN START 


‘Hibernation’ for Blockades B 
and C 
Setting Phase 


Simulation Phase 


REACH FINAL 
WAYPOINT 





Figure 8. | Sequence of states for a blockade-agent. 


As mentioned, the militia guarding the blockade is aggregated with the blockade. 
In this scenario, the militia consists of five guards armed with 100 rounds of ammunition 
each. In the DEFAULT STATE, the blockade possesses a weapon available with 500 
rounds. The probability of hitting the target with a single shot depends on the distance to 
the target (see Table 1). There are three levels implemented. The data entries are based 


on military norms. The probability entries represent one shot per time step. 














Distance to target Probability 
[meters] [percent] 
50 30 
100 20 
200 10 











Table 1. Probability of a successful shot depending on the distance to the target. 
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The convoy can open the blockade with its weapon. In this case, the blockade 
suffers an injury and changes its threat and class level within the state transitions. The 
convoy no longer recognizes the blockade as an attackable enemy. No weapon is 
available to the blockade in the states INJURED and REACH FINAL WAYPOINT. In 
the real world, all blockade militia have been neutralized and the convoy has run through 


the burning tires. 


The numbers of injured blockades acts as a measure of effectiveness (MOE) for 
the examination of the impact of communication. It signifies that the convoy has been 
forced to fight through rather than maneuver around a blockade. 

3. ‘Logic Helper’ 

In order to support the convoy’s decision concerning which route to take through 
the city, the simulation creates ‘logic helpers.’ These helpers recognize an ambush and 
close the street leading into the trap. They give the convoy a kind of human logic 


thinking. 


The helper-agents appear in teams of two, each of which has contrary allegiance. 


A short example serves to illustrate how the helpers act and react (Figure 9). 


a) Initial Scenario b) ‘Logic helpers’ at work c) Final setting 





Figure 9. | Demonstration of the ‘logic helper’-agent. (Best viewed in color) 
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The initial situation at the crossroad (a) develops as follows. The enemy has set 
up an ambush at a four-way-crossroad by blocking three streets. The blockades (red 
crosses)—enemy threat 2—have been detected by the map-agents, which change their 
behavior (yellow crosses). The convoy thus recognizes one street as usable (green 


crosses). 


The ‘logic helpers’ consist of two squads: the blue triangles (b) and the orange 5- 
point-stars. The triangle-agent notices the blockades and shifts into a trigger state in 
which it has a perfect weapon with one round of ammunition. Otherwise, the agent 
leaves the scenario. The agent opens fire only on the nearest 5-point-star-agent, which 
appears in the simulation simultaneously with the triangle-agents. After firing, the 


triangle-agent disappears. 


The 5-point-star-agent can suffer a specified number of hits, depending on the 
street constellation. In this case (b), three hits are necessary to kill the 5-point-star-agent. 
The death of the squad member results in a trigger state transition (SQUAD DEATH), 
where the surviving 5-point-star-agent becomes an enemy threat level 3. Map-agents 
recognize this change and become neutrals, thereby closing the previously unblocked 


street (c). 


From the convoy’s perspective, the final setting develops as follows. The map 
grid friend detects the ‘logic helper’ in its street and becomes a neutral. The convoy will 
get the information about the neutral via communication link. The convoy will then 
know the blockade constellation and the neutral’s location. The neutral does not allow 


the convoy to enter the street leading into the ambush. 


There are a variety of ‘logic helpers’ which use principles similar to those 
outlined above. The trigger state transition of triangle-agents can also be activated by a 
neutral or an enemy threat 3. On the other hand, the surviving 5-point-star-agents can 
mutate to neutrals and close the road directly. When the final setting is complete, the 
helpers, like the blockades B and C, hibernate until the associated blockade wave enters 


the scenario. 
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4, Sensor Grid 

MANA structures communication as a continuous flow of information along the 
link. At each time step the organic and/or inorganic information is transmitted to the 
recipient. In a modern digital data network, information is packed, sent and frequently 
updated. A sensor enters an ‘offline’ mode regarding the communication link between 


updates of the information. 


In order to achieve this behavior, MANA implements a perfect cookie-cutter 
sensor that alternates between two states during the main phase of the simulation. The 
use of a sensor that would frequently misclassify contacts would lead to chaotic situations 
within the scenario. The logical chaining relies on accurate information about detected 
agents, and therefore requires the perfect sensor. Due to the fact that MANA does not 
have a 3-dimensional picture, however, the hovering UAV is placed on the ground and 
can scan through walls. In actual combat situations, the UAV altitude would be chosen 


in order to have maximum view of the ground. 


The sensor-agent’s behavior is visualized as a flow chart in Figure 10. After the 
setting phase (RUN START STATE), the sensor-agent scans the environment within its 
sensor range and enters the information onto its organic map, where it remains for only 
one time step. This data is sent to the convoy’s inorganic map. There is only one chance 
to put the data in the communication queue. In the next step (SPARE 2 STATE), the 
sensor grid goes blind by setting its sensor range to zero. The agent stays in this mode 
for a fixed duration before it automatically falls back into SPARE 1 STATE and receives 


the sensor capability. 
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RUN START No sensor capability 
during scenario setting 


Duration: 100 : 
Setting Phase 


Simulation Phase 


SPARE 1 Sensor capability 
Duration: 1 


Tasks: 
- Updating organic map 
- Passing organic map to the convoy 


SPARE 2 No sensor capability 
Duration: variable 


Figure 10. The sensor behavior. 





The sensor grid is built by five sensors possessing a direct link to the convoy. 
The communication parameters—latency, reliability, capacity, the sensor range and the 
pause between updates (duration in SPARE 2)—vary as factors in the simulation. The 
sensors remain in fixed positions during the simulation. Normally, a sensor grid is a self- 
maintaining system designed to optimize coverage of the area. 

5. Super-Agents 

The scenario contains three squads of super-agents associated with one blockade 
wave. These squads guarantee a random setting of the blockades within the wave. 
Different intensity levels are created by varying the number of super-agents per squad. 


The number of blockades in a specific wave thus becomes a controllable enemy factor. 


A super-agent has a perfect cookie-cutter sensor and a perfect weapon with one 
round of ammunition. The agent reacts to the appearance of its associated blockade wave 
(synchronized by RUN START STATE), chooses its target randomly and kills it. 
MANA excludes the possibility that squad members may shoot at the same target 
simultaneously. After firing, the agent falls back into AMMO OUT STATE, after which 


it disappears from the scenario map. 
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6. Convoy 

The convoy is simulated by one agent. The aggregation, as mentioned before, 
reduces the complexity in order to model the correct convoy behavior of several vehicles 
and their intra-squad communications. The convoy is located at its starting point during 
the setting phase of the scenario but does not move. Fuel status is used as a surrogate for 
urgency in returning to base, since actual fuel requirements are not used in the model. 
The convoy transitions automatically into the REFUEL BY FRIEND state. After 2200 
time steps (real world: about 42 min) it jumps into the FUEL OUT state. This behavior 
change results in a desire to reach the base as quickly as possible. Figure 11 
demonstrates this organic behavior— information gathered by the convoy’s visual 


sensor—by showing the differences between the two trigger states. 
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Figure 11. The convoy’s organic setting in REFUEL BY FRIEND (left) and FUEL 
OUT (right). 


In the beginning of the simulation phase, the convoy attempts to avoid the 
blockades that it has visually detected. The map-agents (Enemy Threat 1) strongly repel 
the convoy when they are in close range. The blockades themselves also have a deterrent 
character within the convoy’s sensor range (400 meters). In this scenario, the agent 
considers only one waypoint to be its final destination—the base. The NEXT 


WAYPOINT draws the agent. 
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When the convoy shifts into FUEL OUT, the final destination becomes a stronger 
focal point while the blocked streets deterrence parameters are reduced. The convoy will 
now run through the blockades in order to reach the secure area. In both states, the 
convoy has a ‘short-distance-weapon’ to open a blockade. The agent can use the weapon 


in a radius of 25 meters around a blockade, and will hit the barrier with certainty. 


Inorganic SA: Inorganic SA: 
Min App. Max. Inf. Min App. Max. Inf. 


EnemyThreat? 0 «{ _{ +!) so 4] 00 4] Enemy Threat1 -15 _ | | 300 4] 
Enemy Threat2 45 «{ J +) 75 4] [300 4] Enemy Threat2 -15 J | p 4 [300 4] 
Enemy Threat3 0 «J _J +!) po =& {10000 | Enemy Threat3 0 a p 4 {10000 | 
Friends i) ae eo o 6 00 4] Friends 40 BE pl o 6 200 4] 
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Figure 12. The convoy’s inorganic settings parameters in the REFUEL BY FRIEND 
(left) and FUEL OUT (right). 
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Similar to the organic parameters, the inorganic parameters change during the 
transition. At the start, the convoy avoids streets blocked by ‘logic helpers’ because of 
the repellant nature of neutrals. In addition, the convoy uses its inorganic information to 
stay away from blockades. The map agents draw the convoy toward usable streets. In 
the FUEL OUT state, the levels for blocked streets are set low (-15) and the friend 
attraction is reduced in comparison to the previous trigger state to support the organic 


desire to reach the final destination. 


E. SCENARIO SCRIPT 
The script of the scenario is visualized in the following figure. The figure 
demonstrates the subdivision—setting and simulation phase—and the associated time 


steps between the events. One purpose of separating the setting and simulation phase lies 
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in the comparability. Using the same seed!3, setting identical levels for the blockades 
and varying only the communication factors result in the same enemy set up. This makes 


a pair-wise comparison of runs possible. 


The events are summarized for the different agents in Figure 13. 


Stetting Phase Time Line 


0 
Convoy is placed and cannot move until settings are completed 


Blockade A and associated ‘logic helper’ are set and remain active 


Blockade B and associated ‘logic helper’ setting 
‘Hibernation’ of Blockade B and ‘logic helper’ 


Blockade C and associated ‘logic helper’ setting 
‘Hibernation’ of Blockade C and ‘logic helper’ 


Simulation Phase 


Convoy starts moving—REFUEL BY FRIEND state 
Sensor grid activated 


Blockade B and associated ‘logic helper’ appear 


Blockade C and associated ‘logic helper’ appear 


Convoy changes trigger state to FUEL OUT 


Time Limit—scenario stops 





Figure 13. The scenario script. 





13 From the MANA Manual p. 14: “Seed: This number is used to generate all the random numbers 
used in a run (start position, hit probabilities etc.).” It was observed that small changes in the terrain file 
and in the numbers of agents placed on the battlefield lead to different blockade settings, even using the 
same seed. 
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F. MEASURE OF EFFECTIVENESS 

MANA generates an EXCEL file that includes several fields, such as ‘Reach 
Final Waypoint,’ run time, number of casualties in each squad and number of injured 
members within a squad. This file contains one row per run. The software package also 
delivers one output file per run which stores all agents’ end state data. This file shows 


the state in which the agent ended its final location and the number of hits. 


The MOEs for the simulation are derived and extracted from this available data 
collection. These MOEs are directly related to the research focus and will be defined 
precisely. They are mutually exclusive in order to avoid overlapping. 


° Communications influence the convoy’s route through the street channels. 
Therefore, the mission length is deduced by subtracting the setting phase 
from the run time. The linear transformation is done in the data post- 
processing. 


e The precision of the information transmitted via the communication link 
leads the convoy to pursue different paths through the city, resulting in 
different numbers of broken blockades. This number is also a MOE, and 
is calculated by adding the number of injured members within the 
blockade squads. 


e Communications affect the number of hits the convoy suffers. A Ruby!4 
script is used to read the data out of the agent end state files. The routine 
opens the file by the predefined filename and reads out the identification 
(run and excursion), the needed information final location and the number 
of hits. 


e In the case that the convoy does not reach its final destination, its last 
position in the scenario leads to conclusions about the cause of its failure. 


An explanation of which MOEs are the most reliable indicators is made in 


Chapter IV, after the analysis of the collected data. 


14 Ruby is an open source programming language which provides excellent scripting and data 
manipulation capabilities. Available at http:/www.ruby-lang.org/en. 
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Hil. DESIGN OF EXPERIMENT 


Combined with an efficient Design of Experiment (DOE), the MANA simulation 
tool provides a good seed for the data farming and promises a successful harvest. This 
chapter focuses on the DOE used to verify the model and gain insights about 
communication aspects in urban terrain. First, the Nearly Orthogonal Latin Hypercube 
(NOLH), the main DOE technique used in this thesis, is introduced together with its 
strengths and weaknesses. Factors for the scenario are divided into two groups, 
controllable and uncontrollable. The design used for each group is explained. In the last 
step, the designs for assessing the model quality and examining communication aspects 


are derived. 


A. THEORY OF NEARLY ORTHOGONAL LATIN HYPERCUBES 

Insight occurs most efficiently in experiments that vary several factors 
simultaneously. One possibility is a full factorial design. Such a design contains all 
combinations between all factor levels. Despite the ready availability of computing 
power today, the number of runs increases exponentially upon the addition of factors and 
levels. In order to examine 4 factors with 10 levels each, 10,000 (= 10*) design points 
would be necessary. This must further be multiplied by the number of replications for 
each design point. The total number of runs quickly reaches millions or more when 
several factors are explored. These kinds of gridded designs are thus useful only for a 


small number of factors and levels. 


Lieutenant Colonel Thomas Cioppa, an NPS PhD graduate, researched in the area 
of experimental design and found a ‘smart’ solution to reduce the number of design 
points while still covering the factor space efficiently. In his dissertation,!> he examined 
the use of Nearly Orthogonal Latin Hypercubes. The main characteristics for a NOLH 


design and the resulting consequences are briefly summarized: 


15 Lt.Col Tomas M. Cioppa, “Efficient Nearly Orthogonal and Space-Filling Experimental Designs 
for High-Dimensional Complex Models,” PhD dissertation, Operations Research department, Naval 
Postgraduate School, Monterey, 2002. 
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The input factors are nearly uncorrelated. Therefore, the design has 
properties that accommodate statistical analysis. Specifically, 
multicollinearity is not a concern when using linear regression. 


The space-filling properties of the design points sample across the input 
space. 


The reduction of design points when compared to a full factorial design, 
does not substantially decrease the capability to analyze many effects and 
interactions. 


A NOLH design opens the opportunity to simultaneously examine a large number 


of input factors. Although this thesis does not focus on the theory and mathematical 


background for NOLHs, it is a useful tool to explore a variety of input factors within a 


reasonably sized experiment. 


A small example, see Figure 14, demonstrates the differences between a 2*- 


factorial design and NOLH-design involving four continuous factors with a range of | to 


10. A low-high (2 level) design is compared to a NOLH design!® with 17 design points. 


The number of design points are nearly identical with 16 (2°) and 17, but the spread 


across the 4-dimensional input space is different, as evident in the pair-wise scatterplots. 
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Figure 14. 


Tt le ie tt leeds al th 
24 6 861012 24 6 861012 24 6 81012 24 6 8101: 24 6 861012 24 6 861012 24 6 81012 24 6 8101: 


Comparison between 2‘-factorial design and NOLH with 4 factors. 


16 This design uses the spreadsheet available at http:/harvest.nps.edu/ <Software Download>. 
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The 2*-factorial design covers only the corners, while the NOLH fills the space 
without touching the extreme points in the corners. Both designs are orthogonal, as 


shown in the correlation matrices (Figure 15). 


k * ° A 
2°-factorial Design NOLH Design 
Correlations Correlations 

Factor1 Factor2 Factor3 Factor 4 Factor1 Factor2 Factor 3 
Factor 1 1,0000 oOo000 o0000 0,000 Factor 1 41,0000 ooo00 00000 
Factor2 00000 1,0000 00000 00000 Factor2 0,0000 11,0000 0.0000 
Factor3 0.0000 00000 1,0000 0,000 Factor3  0,0000 00000 41,0000 
Factor4 00000 00000 00000 1,0000 Factor 4 0.0000 «8600000 o,0000 


Figure 15. Comparison of the correlation matrices of a 2‘-factorial design and NOLH 
design. 


The only noticeable disadvantage of a NOLH design is the lack of coverage at the 
edges. Extreme low and high settings for input factor combination are not achievable 
with this kind of design. The next section, which deals with uncontrollable factors, 


shows a solution to cover the edges. 


B. NOISE FACTORS 

The introduction of uncontrollable factors, also called noise factors, is necessary 
to examine the impact of communication aspects on a broader base. The enemy’s 
behavior and capabilities affect outcomes, but are outside of blue control. Modeling 
them explicitly, but classifying them as noise factors, allows the experimenter to test the 
robustness of proposed solutions. All controllable combinations of the communications 
aspects are tested in several scenarios with different noise factor settings, such as the 


density of blockades. 


The number of blockades per wave is directly controlled by the number of super- 
agents associated with the wave. In a first approach, a NOLH design for the noise factors 
is examined. The use of the NOLH Excel spreadsheet leads to a DOE with 17 design 


points. Table 2 presents the chosen ranges. A large range has been chosen to stress the 
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Factor 4 
06,0000 
0,0000 
06,0000 
41,0000 





communication factors. Although it is clear that 15 blockades, the upper bound, is an 


unrealistically high number, the range is designed to also explore results for a more 


reasonable number of blockades. 




















Number of Agents | Resulting Blockades per 
Wave 
Max Min Min Max 

SuperAgent 11 11 i 1 5 

SuperA gent 22 14 10 1 5 

SuperA gent 33 17 13 1 5 
Wanted Total number’ of 3 15 
Blockades 











Table 2.  NOLH settings for the super-agents and the resulting blockade outcomes. 


The actual resulting range for the total number of blockades is 5 to 12. The 
NOLH-design does not explore the values below five and greater than 12. These ranges 
are also important, however. Therefore, eight new design points are attached: one setting 
for achieving 15 blockades and seven settings to fill the space below five. These seven 
settings are derived by taking one blockade per wave. The complete Excel spreadsheet is 


contained in Appendix Table 4. The distribution of the resulting blockades is shown in 


Figure 16. 


Total Blockades 








Figure 16. The distribution of the resulting numbers of blockades. The marked parts 
are the NOLH-elements. 
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The additional design points adversely affect the orthogonality of the design. As 
visualized in the correlation matrices in Figure 17, the entries above and below the main 
diagonal are about 0.5, which clearly differs from 0. The correlation matrix for the pure 
NOLH is given as a comparison, where the absolute values of the off-diagonal entries are 


negligible. The differences in correlation occur when the numbers are rounded to 


integers. 
Correlations Correlations 
SuperAgent11 SuperAgent22 SuperAgent33 SuperAgent11 SuperAgent22 SuperAgent33 
SuperAgentt 1 41,0000 -0,0805 -0,0085 Super&gent11 40000 0,4956 0,5274 
SuperAgent22 -0,0805 41,0000 -0,0085 Super Agent22 0,4956 41,0000 0,5271 
SuperAgent33 -0,0085 -0,0085 41,0000 Super Agent33 0.5274 05274 41,0000 


Figure 17. The correlation matrix for the pure NOLH (left) and the combined design 
(right). 


The disadvantage is acceptable because this design is crossed with another 
NOLH-design. Each design point is combined with each design point of the other design. 
The correlation within one of the designs does not affect the result of the Cartesianly 
joined design. The resulting combinations are mutually orthogonal. The correlation 


matrix for a crossed design is detailed later in this thesis. 


C. CONTROLLABLE FACTORS 

The controllable factors in this research are parameters that directly or indirectly 
influence the blue force communication capabilities and are controllable by the blue 
forces. This thesis explores seven factors of the scenario. These factors are defined in 


the following section. Attributes for the sensor grid are changed simultaneously for all 
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squads in a lock-stepped mode. Instead of increasing the number of factors, the lock- 


stepping is scripted in the XML!’ study-file created by a special software tool called 


Tiller.!8 


The first six factors of the scenario in question are physical parameters of the 


sensor grid agents. The last factor is a parameter directly affecting the convoy. When 


possible, the MANA definition found in the User’s Manual!? is given or an explanation is 


provided as to how this factor was created in the scenario. 


Latency: (MANA definition) Number of time steps taken for each 
message to reach the squad. 


Reliability: (MANA definition) Likelihood that a given message will be 
successfully sent on a link per try. Repeated attempts are made at 
resending unsuccessful messages until they are — successfully 
communicated. 


Capacity: (MANA definition) Number of messages that can be sent 
through the link per time step. This factor corresponds to the bandwidth in 
real world terms. Due to the workarounds, the number of messages is not 
linear to the number of agents on the battlefield. This factor allows for 
assessment of effects and interactions, but a one-to-one mapping is not 
possible. 


Pause duration between Updating: This factor describes the amount of 
time the sensor grid pauses before sending an update. MANA does not 
provide a direct data entry; therefore, a workaround is created that 
effectively blinds the sensor for the pause duration. 


Communication range: Maximum range between the centroid of the 
transmitting and receiving squads. The communication range as a factor is 
only applied in the design for assessing the model’s behavior, and varies 
in binary style (0 m vs. 2000 m). This factor is used as surrogate for 
whether or not a communication link is available. 





17 The definition for XML comes from www.wikipedia.org: “The Extensible Markup Language 
(XML) is a W3C-recommended general-purpose markup language that supports a wide variety of 
applications. XML languages or “dialects” are easy to design and to process.... Its primary purpose is to 
facilitate the sharing of data across different information systems, particularly systems connected via the 


Internet.” 


18 The Tiller is a software-tool provided by Project Albert to connect the scenario with the DOE. 
Based on the MANA scenario, the input factors and their ranges are entered and the kind of design is 
chosen. The software will automatically create a study including the DOE. This folder contains the 
scenario XML-file, the terrain and the XML-study file. The study file can be used directly by the high 
performance computer. 


19 MANA Version 3.0 Users Manual, D. Galligan, M. Anderson, M. Lauren, July 2004. 
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e Sensor range: (MANA definition) The radius in cells in which an agent 
can detect and classify targets. 


e MaxAge (Inorganic threat Persistence): (MANA definition) The number 
of time steps that must pass before a contact on the squad inorganic 
situational awareness disappears. MaxAge is set at 1.5 * Latency. This 
means that, at the minimum, the contact remains on the map for the delay 
time in the network; the contact may also stay an additional time. When a 
contact is on the map, it may influence an agent’s decision. 


All seven controllable factors of the scenario are designed to provide insights 
about communication aspects in urban terrain. Their input ranges are outlined in the 


Table 3. 






































MANA Scenario Real World 
Lower Upper Lower Upper 
Latency 50 500 6 sec 60 sec 
Reliability [%] 10 100 10 100 
Capacity 300 500 Not directly convertible 
Pause between 20 200 2.5 sec 25 sec 
Updates 
Sensor Range 300 500 300 m 500 m 
MaxAge Lockstep function: 1.5 * Latency 





Table 3. Input factor ranges. 


With the exception of MaxAge, all of these factors are entered in the spreadsheet 
for creating the NOLH design. In this case, a 65 design point NOLH has been chosen to 
achieve a dense spread. The map clearing frequency is added in the spreadsheet later as 


an extra column. 


D. DESIGN FOR ASSESSING THE MODEL QUALITY 
The experiment explained in the following section tests the model in order to 


explore the influence of MANA limitations on the scenario discussed in this thesis. 


To gain insights, two different scenarios were established for this design of 


experiment (DOE). The first scenario excludes the communication aspects between the 
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sensor grid and the convoy. In other words, the convoy has only visually collected 
information available. On the other hand, the convoy can rely on perfect information in 
the second scenario. In this case, perfect information is given by a latency of zero time 
steps, a reliability of 100 percent, capacity of 500 messages per time step, pauses between 


updates of 10 time steps, and a sensor range of 350m for the sensor grid. . 


The DOE is set up by crossing the noise settings (25 design points) with a binary 
communication range for the sensors (Om and 2000m). The change for the 
communication range is lock-stepped for all five sensors in the grid. The DOE conducts 


100 replications for each of the 50 design points. 


There are several ways to eliminate communication in the scenarios. This 
experiment alters the physical settings of the sensor grid instead of disabling the agents 
themselves in the scenario. A change in the number of agents placed on the battlefield 
would result in a different blockade setting for a seed used in the run with and without 
communications. Therefore, the pair-wise comparability would get lost. Although it is 
also possible to set the inorganic desires of the convoy to 0, this method requires several 


changes—specifically, the involvement of 2 trigger states and 3 desires. 


A fixed set of 100 seeds2° guarantees that each seed provides the same blockade 
setting, given the same number of super agents, but independent from the communication 
settings. Because of this fact, runs with and without communication are directly 
comparable in the analysis phase. The paired t-test, the most powerful tool for comparing 


means of two normal samples, is applicable, as Chapter IV will make evident. 


E. DESIGN FOR EXPLORING COMMUNICATION ASPECTS 
This section explains the design to capture the impact of communication factors. 
As mentioned, this test employs a crossed design using the controllable and 


uncontrollable factors. Each design point of the noise factors is tested against each 


20 The Tiller software generates a set of 100 seeds. The set is applied to all design points. This means 
that replication #1 on design point #1 has the same seed like replication #1 on design point #2. 
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design point of the controllable factors. The broad approach ensures that the 
experimenter may distinguish between factors that dominate the scenario and factors that 


can be ignored in further experiments. 


The crossed design consists of 1625 design points, each with 100 replications. 
The correlation matrix in Figure 18 shows that there is no correlation between the noise 
and controllable factors (see box). As a result, useful properties for the analysis are 
provided. This matrix also shows the strong correlation between the factors Latency and 
MaxAge, the linear function of Latency (see oval). Instead of presenting the level of the 
super agents, which are varied, the number of resulting blockades is used to calculate the 
correlation matrix. The number of blockades is achieved by a linear transformation of 
the number of associated super agents. Therefore, the correlations among the blockades 


are identical to the correlations among the super agents. 





Correlations 

Blockade 4 Blockade BBlockade C Latency Reliability PAUSE duration SENSOR Range Capacity MaxAge 
Blockade 4 10000 04956 05271 -0,0000 -0,0000 0,0000 0.0000 + o,0000 0,0000 
Blockade B 04956 10000 05271 oo00 oo00 0,0000 o,0000 Oooo o,0000 
Blockade C 05271 05271 10000 -0,0000 -0,0000 -0,0000 -0,0000 0.0000 -0,0000 
Latency 10000 -0,0013 0,0008 0.0005 -0,0008 
Reliability -0,0013 = 1,0000 -0,0031 00006 oo007 -0,0013 
PAUSE duration 00008  -0,0031 41,0000 0,0003 0.0001 0,0009 
SENSOR Range 0.0005 + 0,0006 0,0003 41,0000 -0,0002 o,0004 
Capacity -0,0008 0,0007 0,0001 -0,0002 71,0000 #8 -0,0006 
MaxAge 14,0000 -0,0013 0,0009 0.0004 -0,0008 41,0000 





Figure 18. The correlation matrix for this communication aspect design. By crossing 
the two designs Cartesianly, orthogonality is obtained between 
controllable and uncontrollable factors, even when there is correlation 
among factors within each design. 
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IV. DATA ANALYSIS 


This chapter focuses on data processing and data analysis. It provides 
information about the software package that is used and statistical tools that are applied to 
gain insights from the model. The next section describes how the raw data is cleaned up 
and prepared for the analysis. In the first analysis part, two important questions are 
raised: 

e How well is the model doing? 

e How reliable is the model in examining the focus of the research? 

The answers allow drawing conclusions about the usefulness of modeling a 
scenario like this in MANA by choosing an approach similar to that of the author. 


Finally, the main experiment is analyzed and the results are presented. 


A. STATISTICAL SOFTWARE PACKAGE JMP 

To conduct an analysis on a large data set a professional statistical software 
package is necessary. The author considered several packages and decided to use JMP 
The Statistical Discovery Software/™. In the manual, the software developers state that 
JMP is easy to learn.”! The author agrees on this point and is convinced of its user 
friendly appearance. Some advantages experienced while working with JMP are briefly 


: 22 
summarized: 


e GUI interface for all activities. 

e Spreadsheet features for manipulating large data sets. 

e A broad range of graphical and statistical methods for data analysis. 
e Capability of grouping data and computing summary statistics. 

e Tools for printing and moving analysis results between applications. 


JMP provides the entire tools necessary for the discovery tour through the data sets in 


this thesis research. 


21 JMP The Statistical Discovery Software, Introductory Guide Version 5. 
22 JMP The Statistical Discovery Software, Introductory Guide Version 5. 
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B. RAW DATA POSTPROCESSING 

The raw data returned by the Maui High-Performance Computer Center 
(MHPCC) consists of two types. One is an EXCEL file containing the factor levels and 
various squad outputs, e.g., the number of casualties or injuries. This file can directly 
import into JMP and be manipulated within the software tool. The other data collected 
are the agents’ end data for each run. In order to extract the main entries, a small Ruby 
program (see Appendix Figure 36) was written and applied. This tool reads in each file 
and puts the important facts, i.e., number of hits and final location of the convoy, in an 


extra text file. 


The JMP software package has good spreadsheet characteristics, as mentioned in 
the previous section. In the next step the columns of the csv file and the columns of the 
text file are joined. However, only a few of the more than hundred columns are 


interesting and necessary for this research thesis. Hence, the others are deleted. 


Data entries are converted and the following new columns are added in order to 
achieve more readability. 


e New column: The number of super agents per wave is translated into 
blockades per wave. 


e New column: The mission length is calculated by subtracting 100 time 
steps from the scenario run time due to the setting phase which was 
explained in Chapter II. 

° New columns: All time step measures are converted into seconds. This is 


done for all factors and results. 
As mentioned in Chapter II, the MOEs are the mission length, the number of hits, and the 
number of broken blockades. The last one is the sum of three columns, specifically the 


number of injured blockades. Also calculated is the total number of blockades. 


C. STATISTICAL METHODS 
After preparing the data set the tools used to gain insights are briefly introduced. 
The analyst’s tool bag consists of graphical tools, parametric and non-parametric 


procedures, multiple regression, and regression trees. 
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1. Graphical Analysis 

A commander believes more of what he sees than what he is told. Following this 
common staff slogan, the data is analyzed with simple graphical tools, e.g., histograms. 
Graphs and plots provide the first impression about some insights and ease the 
interpretation and explanation, especially to higher ranking officers without detailed 
knowledge about analysis techniques. 

2 Parametric and Non-parametric Procedures 

Parametric procedures are based on distributional assumptions. The most common 
parametric tests require normality, i.e., nice bell-shaped behavior in the data or at least for 


the estimators. On the other hand, non-parametric procedures are often referred as 
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“distribution-free.””~ The data analysis in this research uses the paired t-test as a member 


of the parametric procedures and the Shapiro-Wilk W-test from the non-parametric 
procedures. 


e The verification experiment delivers paired data. The paired t-test is 
conducted to investigate whether the two samples could have the same 
mean. The test requires independent and normally distributed sample 
means.*4 Fortunately, the t-test is robust against non-normally distributed 
data for large sample sizes. The paired t-test focuses on means. The 
Central Limit Theorem states that the sample mean is approximately 
normally distributed given a sufficiently large sample size.?5 


e The Shapiro-Wilk W-test2° is a “distribution-free” procedure. The 
Shapiro-Wilk W-test examines the hypothesis that the sample comes from 
a normal population. 


3. Regression Trees 
A regression tree is a Data Mining’ technique. “There is no implicit assumption 


that the underlying relationships between the predictor variables and the dependent 


23 Jay L. Devore, “Probability and Statistics,” 6th Edition, 2004, p. 667. 
24 Jay L. Devore, “Probability and Statistics,” 6th Edition, 2004, p. 381. 
25 Jay L. Devore, “Probability and Statistics,” 6th Edition, 2004, p. 239. 


26 W.J. Conover, “Practical Nonparametric Statistics,” 3rd Edition, 1999, p. 450. If the test statistics 
W “is close to 1.0 the sample behaves like a normal population.” If W “is too small and far below 1.0, the 
sample looks non-normal.” 


27 David Hand, “Principles of Data Mining,” 2001, p. 2, Definition: “Data mining is the analysis of 
(often large) observational data sets to find unsuspected relationships and to summarize the data in novel 
ways that are both understandable and useful to the data owner.” 
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variable are linear, follow some specific non-linear link function, or that they are even 
monotonic in nature.”28 More advantages of this assumption-free tool are listed in the 


JMP help as follows: 


e “it is good for exploring relationships without having a good prior model 
e it handles large problems easily, and 
e the results are very interpretable.”29 


The regression tree method is applied to a continuous response variable and input 
factors which can be continuous or categorical. The tree structure is built by binary 
recursive partitions. The goal is to minimize the sum of squared deviations within the 
separated groups. This process increases the “purity” in the class.39 The sequence of 
factors involved in partitioning allows drawing conclusions on their importance. 
Consecutive splits in a branch where different factors are involved may indicate an 
important interaction. Another combination, i.e., one factor showing up twice in a row 
may be an indication of a non-linear effect. The analyst gets an idea on a potential linear 
regression model. On the other hand, factor levels are noticeable when a split occurs. In 
some cases, when a significant improvement is noticed, this factor level can be 
interpreted as a threshold. 

4. Multiple Linear Regression 

“Regression analysis is one of the most widely used techniques for analyzing 
multifactor data.”3! In this thesis, multiple linear regression analysis is applied. It deals 
with one continuous response and continuous and categorical input factors, called 
predictors or regressors. The response is a linear combination of coefficients of predictors 
and an additive statistical error. This error is often assumed to be uncorrelated and 


normally distributed with mean 0 and constant variance. 





28 StarSoft Technologies, www.statsoft.com/textbook/stcart.htm, December 2006. 
29 JMP 5.2.1 online-help Index “Partition.” 
30 David Hand, “Principles of Data Mining,” 2001, p. 343. 





31 Douglas C. Montgomery,” Introduction to Linear Regression Analysis,” 3rd Edition, 2001, p. xiii. 
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Regression models are useful to describe data, estimate parameters, predict and 
estimate outcomes. Of course, agent-based models are not built for prediction purposes, 
as the author mentioned in Chapter II. In this thesis, regression models are used to gain 


insights about the important factors and their interactions. 


D. ANALYSIS FOR ASSESSING THE MODEL’S QUALITY 

iF Unfinished Runs 

One technique to consider on whether this model is doing a “good” job is a look 
at the runs and which of them are terminated by the time limit. Occasionally, the convoy 
gets “stuck” along the way. This would never happen in the real world. Within the data 
set of 5000 runs, only in 13 cases (0.28 %) did the convoy not reach the final 
destination—that is, it got stuck on its way. Figure 19 shows the distribution of the last 


location on the map as a function of the communication range. 














CommRange=0 CommRange=2000 
Distributions Distributions 
FinalLocation FinalLocation 
Frequencies Frequencies 

Level Count Prob Level Count Prob 

529-625 1 014286 288-578 2 033333 

632-134 4 057143 377-838 1 016667 

644-271 2 0.28571 §54-433 1 O,16667 

Total 7 1,00000 588-492 1 016667 

529-625 632-134 644-271 N Missing in) 641-114 1 016667 

3 Levels Total 6 1,00000 

N Missing 0 














5 Levels 


Figure 19. Distribution of the final locations split by the communication range. 


The author re-ran some scenarios and noticed that these final locations result from 
terrain issues. Mostly, the terrain around the last location is not smooth enough to allow 


the convoy to move on. 


Due to the fact that at most two runs per design point are terminated by the time 
limit, the author evaluates their occurrences as non-critical and leaves them within the 


data set. 
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2. MOE Selection 

In this first approach, the author decided to get a broad overview on the 
simulation outcome by looking at the combined correlation matrix, including the input 
factors and MOEs. The first question is whether communication has any impact on the 
outcome. In this experiment, the communication range has two levels, 0 and 2000, which 
can be translated as having and not having communication. In the former case the 


convoy can take advantage of getting perfect information without latency. 


The blue box in Figure 20 indicates that having communications increases 
mission length (positively correlated), decreases the number of broken blockades and the 
number of hits (both negatively correlated). The way communication affects the outcome 
fulfills the author’s expectations. The levels do not show a strong correlation (near -1 or 


1), but they differ noticeably from zero. 


Correlations 

Blockade 4 Blockade B Blockade C CommRange Mission Length Broken Blockades #HITS 
Blockade 4, 4 ,0000 0,4956 052741 0,0000 0.1133 0.2424 0,4205 
Blockade B 0,4956 41,0000 05271 60,0000 0.2141 02224 04381 
Blockade C 05271 0,5271 41,0000 -0,0000 0,2024 0.2546 0.3955 
CommRange 0,0000 0.0000 = -0,0000 41,0000 02088 -0,1348 -0,1980 
Mission Length 0.1133 0.2141 0.2024 41,0000 0.2291 0.3915 
Broken Blockades 02424 02224 0.2546 0.2291 4 ,0000 0,5024 
#HITS 0.4205 04381 0.3955 0.3915 0,5024 4 ,0000 





Figure 20. This correlation matrix shows the impact of having communication for the 
MOEs (blue box) and the correlation among the MOEs which makes a 
pre-selection possible. (Best viewed in color) 


The red box in Figure 20 demonstrates the correlation among the MOEs. 
Obviously, the three MOEs are correlated. The correlation between “Broken Blockades” 
and “#Hits” reaches 0.5. This is meaningful due to the fact that fighting through the 
blockades is only possible for the convoy within a 25 meter radius of the blockade. This 
is the range in which the blockade has its highest probability of hit. The other 
correlations, including with mission length, are clear, but not so strong. However, all 


correlations are positive. This means that examining one MOE deeply provides 


40 


information about the other two MOEs. In this case, the author’s focus lies on the 
number of hits the convoy gets. Nevertheless, the analysis on the other two MOESs is also 
conducted. The results, only briefly mentioned here, are not consistent. In a leap ahead, 
the p-values for the paired t-test on the two samples for each MOE do not stay below the 
given level of significance of 0.05. This means that at some design points the means for 
one MOE with and without communications are different, but they are not statistically 
different. There remains the chance that both samples are from populations with the 
same mean at the given design point. The most dangerous elements in combat situations 
are the bullets. In the author’s opinion, the number of hits is the most important measure, 
which is also the only statistically consistent MOE in this simulation. 

3. “Measuring” the Impact of Communications 

This section deals with the question of how well the model behaves. The 
experimental setup allows comparing two runs with the same blockade setting and 
different communication parameter settings. Referring to the MOE selection, the author 
uses only the number of hits for examining the impact of communication. Within JMP, a 
data table is created that contains the needed parameter information and the number of 
hits with communication and the number of hits without communication in one row. In 
the next step, the differences between the number of hits without and without 
communications are calculated. A negative value means that the convoy with 
communications suffers more hits than the one without communications. In addition, 
columns are attached containing binary indicators of whether the runs with 
communication ended with a lower or equal number of hits. Instead of regarding each 
run, a summary table is created which contains the parameter setting, the mean of the 


differences, and their standard deviations for each design point. 


At first, the variability is checked with a box plot containing all design points. 


This is done in Figure 21 below. 
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Variability Chart for Hit Difference 
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Figure 21. The variability for each design is demonstrated in this range plot. 


The extracted data contains a lot of variability. Especially, the large negative 
amplitudes are an indicator that the situation is inherently highly variable. Agent-based 
models are stochastic models and therefore they incorporate randomness. Small negative 
amplitudes could have been caused by this effect. All 25 means are positive and with 
more blockades showing they are clearly distinguishable from zero. Under the 
hypothesis that communications have no effect on the outcome, the chance of observing 
fewer hits (i.e., positive median hit difference) at all design points, has the probability of 
(4) (=3*10°) if the design points are uncorrelated. An empirical test shows that the 
correlations among these 25 samples are consistent with correlations among 25 normally 
distributed random samples. Therefore, on the whole, communications help the convoy, 
especially with increasing difficulty level. Some scenarios were re-run in order to find 
the true story behind these large negative differences. The main causes are summarized: 


e The convoy with communication gets stuck in a position where it is 
exposed to hostile fire. The convoy without communications does not 
stop due to its different desires and route. 


e The occurrence of an unlucky event is also observed. The convoy with 
perfect knowledge avoids blockades in the first wave but it is caught by an 
ambush in the second wave while the convoy without communications 
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gets hits but luckily its later route is safer without any ambush situation. 
Of course, this could happen in the real world. 


° In some cases the convoy with additional inorganic information does not 
behave like a human being. Driven by its weights it ignores the grid 
friends’ and helpers’ signals and runs straight into the blockade, or, even 
worse, into an ambush where fire hails on it. 


The first analysis is an eye opener of the model’s performance. It is obviously not 
perfect, but the question is how “good” or useful is it. This question is examined by 
taking a closer look at the means. The means and the 95%-confidence interval for all 25 


design points are shown graphically in Figure 22. 
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Figure 22. The means of the differences and 95%-CI for all 25 design points 
(#combination). The blockade setting is listed beside [A-B-C-(Total)]. 
(Best viewed in color) 


The graph indicates the main trend that an increasing number of blockades results 
in larger differences. The first box shows that 1 blocked street in wave A (combination 3) 
has a larger impact on the difference than the two previous ones due to a lack of 
alternative routes at the starting point. The drops (11 and 15) occur because the number 
of blocked roads (wave A) decreases in comparison to the previous design points. This 


indicates that the number of blocked streets at the beginning strongly influences the 
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outcome. The density of blockades in the first wave is higher. In order to illustrate this 
result the same plot is shown after reordering the design points within the number of total 
blockades (see Figure 23). The ordering can be interpreted as difficulty levels within one 
group of design points having the same number of resulting blockades. Looking at the 
combinations 23 and 24, it can be concluded that the setting 4-5-3 is harder than 4-3-5. 


The argument holds that the blockade density in wave B is higher than in wave C. 
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Figure 23. Reordered plot showing the means of the differences and 95%-CI for all 
25 design points (#combination). Means in the same group are connected. 
The blockade setting is listed beside [A-B-C-(Total)]. (Best viewed in 
color) 


All of the means on the differences are positive. Having communications reduces 
the expected number of hits. If not, the chance of seeing this is a rare event. This implies 
that the model does a “good” job on average. To gain more insights, the proportions of 
runs which do better with communication are examined. The graph in Figure 24 
demonstrates the proportions for each design point. The ordering of the design points 


remains as found in the previous section. 
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Figure 24. Comparison between runs with and without communications showing the 
percentage of how many runs have less hits, equal number of hits and the 
combined measure. The blockade setting is listed beside [A-B-C-(Total)]. 
(Best viewed in color) 














Two main trends are observable in Figure 24. The green line (a) describes the 
proportion of runs with the same outcome. In a low threat scenario, e.g., for one 
blockade model, most runs have the same outcome. Increasing the threat level causes the 
equal outcomes to decrease. On the other hand, the proportion of runs where 
communications have a strictly positive influence (0) on the result is increasing at the 


same time. The sum of both (x) has been chosen as a measure of the model quality. 


Due to the high variability, the performance drops down below 70% and stays in 
this region. Two dips in this nearly horizontal line are remarkable—in fact the 
performance decreases below 60%. The points have one common pattern. At the 
beginning the convoy experiences a low intensity phase which is followed by a massive 


appearance of blockades in the wave B. 


The lack of performance will be discussed again at the end of this Chapter when 


the author summarizes the findings and evaluates the quality and usefulness of the model. 
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After the graphical examination of the data, a sequence of statistical tools is 
applied to analyze whether the two samples (with and without communication) are from 
different populations. The test procedure is conducted on one design point for 


demonstration purposes. The entire results are listed in the Appendix Figures 37 to 43. 


The distribution of the hit differences are created for each design point in JMP. 
As mentioned before, the paired t-test can be applied even when the underlying 
distribution is not bell-shaped normally distributed. A large sample size guarantees, by 
the Central Limit Theorem, that the mean is roughly normally distributed. This allows 


using standard techniques to examine the mean. 


The author decided to check on normality as well before using the t-test. 


Therefore, a fitted normal distribution provided by JMP is included (see Figure 25). 


Blockade A=1, Blockade B=1, Blockade C=1 
Distributions 
Hit Difference 
Fitted Normal 
Parameter Estimates 


Type Parameter Estimate Lower 95% Upper 95% 
Location Mu 8,29000 452539 12,05461 


-60 -40 -20 0 20 40 60 80 100120 





—— Normal(8,29,18,9728) 


Figure 25. The distribution of the hit difference for a specific blockade setting and its 
fitted normal distribution. 


The distribution seems be bell-shaped around the mean, but the asymmetric 
behavior towards the edges allows the conclusion that this data is not normally 


distributed. To confirm this observation a Quantile-Quantile plot (QQ-plot) 32 is 





32 Douglas C. Montgomery, “Introduction to Linear Regression Analysis,” 3rd Edition, 2001, p. 192 
“Normal Probability plot... . If the sample observations are in fact drawn from a normal population with 
mean p and standard deviation o, the points should fall on a straight line with slope o and intercept p. Thus 
a plot for which the points fall close to some straight line suggests that the assumption of a normal 
population distribution is plausible.” 
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examined. Normally distributed data should show up along a straight line. In a parallel 


approach the Shapiro-Wilk-test is conducted (See Figure 26). 


Quantile Plot Goodness-of-Fit Test 
Shapiro-vvilk VV Test 

Ww Prob=/¥ 

0.833173 =.0001 


Normal Quartile 





-60 -40 -20 0 20 40 60 680 100120 


Figure 26. Qantile-Quantile plot showing that the distribution is definitely not 
normal. This result is confirmed by the Goodness-of-Fit test shown in the 
box. 


Obviously, normality cannot be assumed looking at the plot and the low p-value 
(Figure 26 red box) provided by the Shapiro-Wilk-test. But the absence of normality 
does not mean that no further analysis is possible. The t-test has robustness against non- 
normality, as mentioned in the previous section. A large sample size is necessary to 
apply it and get reliable results. One hundred runs per design point guarantee a 
sufficiently large sample size. Actually, due to the sample size, this can be seen as a z- 
test. JMP provides a GUI to conduct the t- test. The hypothesis under which this paired t- 


test is conducted is stated in the following way: 


A, : HnitDifference = 0 


H, : HaitDifference = 0 


Because of the author’s intent to work out the improvements by having inorganic 
information, a one-sided test is preferred and seems to be more appropriate to answer the 
research question. The result provided by JMP is shown in Figure 27. 


AT 


Test Mean=value 


Hypothesized Value 0 

Actual Estimate 8,29 

dt 99 

Std Dev 18,9728 
t Test 

Test Statistic  4,3694 

Prob = {t| <,0001 

Prob =t 4 ,0000 


Figure 27. The test result for the paired t-test demonstrates that communications 
reduces the number of hits. 


The p-value is smaller than 0.001. This implies that the hypothesis must be rejected. 
Actually, communications significantly reduce the expected number of hits which is 
observed by positive differences. The paired t-test is the most powerful test on two 
sample means. This sequence of tests is also conducted for the other 24 design points 
and is shown in Appendix Figures 37 to 43. The results are similar to the explained one. 

4. Discussion and Conclusions 

Looking at the convoy’s behavior with and without communications it seems that 
the received inorganic information has an impact on the expected number of hits. Having 
support by a sensor grid implies a reduced number of hits on average. The statistical 
analysis shows that for all design points both samples have different means. The model 
does the right thing on average. Its weakness is expressed in the proportion of runs 
where communications does not positively influence the outcome. On average 30% to 
40% (maximum 44%) of the runs ended with a less hits when the convoy does not have a 


sensor grid support. 


This portion sounds high. However, human decision making does not lead 
automatically to the optimal solution. Sometimes a decision has to be made within 
seconds. The convoy leader is under combat stress and has time limitations for his 
decision. In situation like this a human decision-maker relies on heuristics and biases.33 
The perceived information is processed and one or more hypotheses are generated and 


tested. If one seems adequate it is chosen as a possible course of action. Furthermore, 


33 Christopher Wickens, “An Introduction To Human Factors Engineering,” 2nd Edition, 2004, p. 162. 
48 


the humans tend to stick with their first hypothesis. Therefore, alternatives are 
underutilized and rejected. This behavior can result in non-optimal decisions. In some 
cases they are fatal and lead to a large number of causalities. Also, just by chance (i.e., 


bad luck) a good decision can lead to a worse outcome than a bad decision. 


The author is aware of the high variability incorporated. The chance of non- 
optimal outcomes in a decision-making process is smaller than demonstrated by the 
model. With this knowledge, the following analysis has to be interpreted with caution 


and the resulting numbers have to seen more as directions than precise thresholds. 


E. ANALYSIS ON COMMUNICATION ASPECTS 
This section focuses on the analysis of the main experiment examining the 
communication factors. The post processed data set (162,500 rows) is large. The 
variability within the set is enormously high. Regression analysis on the original data 
leads to small R-square values, which is a measure how much variability is explained by 
the regression model. Therefore, the author decided to use the summary data. The data 
is grouped by design points. The means and standard deviations on the MOEs are 
calculated and represented in columns. The data set still contains 1,625 elements. The 
author excluded the factor MaxAge because it is linearly dependent on Latency. A pair 
of linearly dependent regressors causes multicollinearity in the regression model; this 
would degrade the quality of the regression model. 
1. Pre-Analysis 
This section focuses on two pre-tests. Similar to the previously conducted 
analysis, the runs are examined in the case where the convoy did not accomplish its 
mission. Following this, the correlation among the MOEs is analyzed. 
a. Unfinished Runs 
The unfinished runs are extracted. Their distribution is shown in Figure 


28. 
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Final Location 





Figure 28. The distribution of the final location. 


The distribution chart shows that 24 locations are involved. The author re- 
ran some of them. One location has a remarkable count. Location 644-271 (red box) 
contains 88 runs with one common seed. These runs are unfinished due to terrain issues. 
Following the argumentation in section D1, the convoy gets stuck. The surrounding 
terrain is not smooth enough to allow the convoy to move on. 

b. MOE Selection 

The correlation among the 3 MOEs is checked in advance in order to 
reduce the number of MOEs to be tested. The correlation matrix below (Figure 29) 
demonstrates that in this experiment a very strong positive correlation exists among all 


MOEs. 


Correlations 
Mean(Mission Length (real)) Mean(Blockades Broken) Mean(#HITS) 


Mean(Mission Length (real)) 41,0000 0.6631 0,7190 
Mean(Blockades Broken) 0.6631 41,0000 0.9431 
Mean(#HITS) 0,7190 0,9431 41,0000 


Figure 29. The correlation matrix shows that the MOEs are strongly correlated. 


In comparison to the previous experiment, these correlations are powerful 
(near 1). Being consistent, the number of hits is chosen as the main MOE. However, in 
this case, the author did not do any analysis on the rejected MOEs. 
2. Regression Tree with Noise Factors and Communication Factors 
In this first approach a regression tree is used to gain insights into the 
relationships between the factors. The regression tree does not require any assumptions 


in order to be applied. The tree structure is shown in Figure 30. Additionally, the factors 
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involved in splitting are repeated in order to increase the readability. In this step the split 


levels and resulting group means and standard deviations are less important than the 


factor itself. 
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Figure 30. Regression tree on the Mean(#Hit) with all input factors. The first 12 


steps are noise factor splits. The first communication factor that shows up 
is Latency (see red box). 


The presented regression tree consists of 13 splits. At the top the “root” of the tree 
contains all 1625 observations. The first 12 partitions involve exclusively the noise 
factors. It can be concluded that the blockades (i.e., the enemy behavior), are the major 
driving factors in this simulation. Looking at the sequences, it becomes obvious that the 
blockades have a non-linear impact on the outcome. Also, there are most likely 
interactions among the noise factors. The first communication factor appears in the form 
of Latency, at the step 13. This indicates the importance of Latency in the simulation. 


That is, the convoy needs timely information. 


These findings are used to frame the next analysis part, where a regression model 
is derived that includes quadratic terms and two-way interactions among all predictors. 
3. Full Regression Model 
To achieve a full regression model including all statistically significant terms, 
JMP is used. The full model provides insights into whether the underlying assumption of 
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additive normally distributed. Another test will be conducted to reduce multicollinearity, 
which could occur because of the correlation among the noise factors. The stepwise 
regression function in JMP allows adding and removing terms at each step. While doing 
so, the order of the factors entering the model can be observed. This is helpful to see 


which ones are more important. 


The ratio of the variation explained by the model to the total variation included in 
the data is called R’, the coefficient of determination.34 The model shown in Figure 31 


has a high R’ value of 0.9752. 


Response Mean(#HITS) 
Summary of Fit 
RSquare 0.975171 
RSquare Adj 0,97483 
Root Mean Square Error 2862534 
Mean of Response 2854883 
Observations (or Sum Wgts) 1625 


Analysis of Variance 


Source DF Sumof Squares Mean Square F Ratio 
Model 22 51557199 23435,1 2859,996 
Error 1602 13126,95 82 Prob>F 
C.Total 1624 §28698,94 0,0000 


Parameter Estimates 











Term Estimate Std Error tRatio Probe>tt] VIF 
Intercept -4,448744 065584 -678 <0001 : 
Blockades 4. 58087652 0,06438 90,23 0,0000 | 2,0857967 
Blockades B 46175575 0,062567 73,80 0,0000 | 1,9699865 
Blockades C 2,8279617 O,066547 42,50 <,0001 | 22286196 
Latency (real) 01175132 0,004492 2616 <,0001 | 1,0003076 
Pause (real) 0,0265991 0,011119 2,39 0,0169 | 1,0007104 
SENSOR GRID Reliability -0,015095 0,002689 -5,61 <,0001 |1,0001897 
(SENSORRangeC—C“‘“‘“S*S*™C*™*™C™C™C™C:C~ OOS. —«-0.88 0.3801] | 10000306 
SENSOR GRID Capacity -0,003143 0,000969 -3,24 0,0012 |1,0001391 
(Blockades 4-2,32)*(Blockades B-2,32) 0,2400935 0,054652 439 <0001 | 3,3624782 
(Blockades 4-2,32)*(Blockades C-2,32) 01577005 005329 296 0,0031 |3,0786625 
(Blockades 4-2,32)*(Latency (real)-33,0154) 0,0325866 0,003318 9,82 <0001 |1,3847628 
(Blockades 4-2,32)*(SENSOR GRID Reliability-55,0154) -0,004384 0,001688 -2,60 0,0095 |1,0000141 
(Blockades B-2,32)*(Blockades C-2,32) -0,467604 0.085035 -719 =,0001 | 43276564 
(Blockades C-2,32)*(Latency (real)-33,0154) 0,0085581 0.003318 258 0,0100 |1,3847487 
(Latency (real)-33,0154)*(SENSOR GRID Capacity-375,015) 0,0001423 0000057 2,51 00,0121 |1,0390151 
[ (Pause (real)-13,2154)*(SENSOR Range-462,508) 0,0005381 0,000149 3,62 0,0003)| 1,1091384 
(Pause (real)-13,2154)*(SENSOR GRID Capacity-375,015) -0,000606 0,000201 -3,02 0,0026 | 1,1401537 
(Blockades A4-2,32)*(Blockades 4-2,32) -0,443961 0.045012 -986 <0001 | 21648379 
(Blockades B-2,32)*(Blockades B-2,32) 0,5242386 0,063046 832 <,0001 | 4,2470694 
(Blockades C-2,32)*(Blockades C-2,32) 0,3164313 0,046332 683 <,0001 | 2,2936728 
(Latency (real)-33,0154)*(Latency (real)-33,0154) -0,003095 0,000378 -818 <,0001 | 14081417 








(SENSOR GRID Reliability-55,0154)*(SENSOR GRID Reliability-55,0154)  0,0003798 0.000131 290 00037 | 1,324107 


Figure 31. The full regression model. 


The fit seems to be good. But a high R’ value does not automatically guarantee a 
good model. Over-fitting, having too many factors, reduces the usability for predicting 


and interpretability. This will be examined in the next section. To reach this high R? 


34 Douglas C. Montgomery, “Introduction to Linear Regression Analysis,” 3rd Edition, 2001, p. 39. 
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level, 22 predictors plus the intercept term are necessary. The model still includes terms 
which do not influence the outcome significantly. For example, the senor range (Range) 
has a p-value of 0.38. This would normally exclude the term; but, an interaction term 
involving the range is significantly important (see connected red boxes). Good regression 


modeling does not allow taking out main effects that are involved in an interaction. 


The JMP software package provides a test on multicollinearity. The Variance 
Inflation Factor (VIF) is automatically calculated for all regressors. “The VIF for each 
term in the model measures the combined effect of the dependences among the regressors 
on the variance of the term.”35 VIF values exceeding 5 or 10 are an indication of 
multicollinearity. VIF also shows the involved factor. In this case, all VIF are below 5, 
thanks to our efficient design. Multicollinearity seems to be no issue. The author ran 
another small calculation to cross check this result. The correlation matrix including only 
the regressors is used.26 The determinant of the correlation matrix has the range between 
0 and 1. The result for a matrix including at least one pair of linearly dependent 
regressors is 0. On the other hand orthogonal regressors are indicated by a value of | for 
the determinant. The correlation matrix (see Chapter 3 Figure 18) has a determinant of 
0.47. The loss of orthogonality is obvious, but the distance from 0 is large enough to 


confirm the JMP results. 


In the next step, the error term is examined. The differences between the fitted 
regression and the observed values are called residuals. Their distribution is shown in 


Figure 32. 


35 Douglas C. Montgomery, “Introduction to Linear Regression Analysis,” 3rd Edition, 2001, p. 337. 
36 Douglas C. Montgomery, “Introduction to Linear Regression Analysis,” 3rd Edition, 2001, p. 343. 
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Distributions 
Residual MeanHITS) 





Fitted Normal 
Parameter Estimates 


Type Parameter Estimate Lower 95% Upper 95% 
Location Mu 0,000000 -013834 = 0138336 
Dispersion Sigma 2.843078 2.74858 2,944350 


Goodness-of-Fit Test 
Shapiro-Vvilk V¥ Test 


Wo Prob<vW 
0,990708 <,0001 


Normal Quantile Plot 




















—— Normal(? 6e-14,2,84308) 


Figure 32. The residual distribution with fitted normal curve and the result of the 
Shapiro- Wilk W-test. 


The residual distribution has a normal bell shape. Looking at the QQ-plot above, 
the endings show a slightly unusual behavior. Even with a high W value (near 1 means 
normality), this behavior causes the rejection of the hypothesis that the residuals are 


normally distributed. 


Knowing that a main regression testing assumption is not strictly met, the 
robustness of regression against slight non-normality is used. The interpretation of the p- 
values whether a predictor stays in the model or not, has to be done more carefully due to 
the slightly non-normal behavior of the error terms. However, this does not impact the 
significant digits of the p-values. 

4. Reduced Regression Model 

The section deals with the question of how to reduce the number of predictors 
while still maintaining a good model fit. During the stepwise regression the author noted 
the improvements in the R* value whenever a new predictor was added. The graph in 


Figure 33 shows the improvement curve as a function of added regressors. 
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1) Blockades A 
Overlay Plot 2) Blockades B 
3) Blockades C 
1 4) Latency (real) 
5) Blockades A *Latency (real) 
0,95 6) Latency (real)*Latency (real) 
7) Blockades B*Blockades B 
og 8) Blockades A*Blockades A 
o 9) Blockades C*Blockades C 
a O,85 10) Blockades B*Blockades C 
5 11) Reliability 
a 08 12) Blockades A*Blockades B 
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O75 14) Capacity 
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Of? 16) Reliability*Reliability 
17) Range 
0,65 18) Pause (real)* Range 
12345 6 7 6 910111213141516171819 2021 22 19) Latency (real)*Capacity 
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21) Blockades A*Reliability 
22) Blockades C*Latency (real) 





Figure 33. R° improvement by adding a new predictor. 


Looking at the chart, it is remarkable that Blockades A, B and C as main effects 
explain 95% of the variation in the data. This result corresponds to the regression tree in 
the previous section where the first 13 steps created an R’ of about 95%. Again, Latency 
appears as the first communication factor. The next set of terms are the interactions 
between Blockades A and latency and the quadratic terms of Latency, Blockades B, A 
and C. With the 8" element an R* of 97% is reached. The added improvement by adding 
more predictors is minimal. Therefore, the author decided to drop the elements appearing 
after the 8" prdeictor and ignore these terms. The resulting model still explains 97.1% of 


the variation. The reduced model is shown in Figure 34. 


ao 


Response Mean(HITS) 
Actual by Predicted Plot 








Summary of Fit 

















RSquare 0.970951 
RSquare Adj 0.970807 
Root Mean Square Error 3082834 
Mean of Response 28 54883 
Observations (or Sum VYgts) 1625 
Analysis of Variance 
Source DF Sumof Squares Mean Square F Ratio 
Model 8 513340,69 64167,6 6751 ,737 
0 10 20 30 40 50 60 70 80 Error 1616 15358 24 95 Prob=F 
Mean(#HITS) Predicted PO,0000 RSq=0,97 C. Total 1624 528698,94 0,0000 
RMSE=3 0828 
Parameter Estimates 
Term Estimate Std Error tRatio Prob=tt| 
Intercept -5,654431 0,261101 -21,66 <,0001 
Blockades 4, 5,7354801 0,065913 87,02 0,0000 
Blockades B 46237153 0,063442 72,88 0,0000 
Blockades C 2,9598465 0,064269 46,05 <,0001 
Latency (real) 04175985 0.004837 2431 <,0001 
(Blockades 4-2,32)*(Latency (real)-33,0154) 0,0371252 0,003037 1223 <,0001 
(Blockades 4-2,32)*(Blockades 4-2,32) -0,261045 0,03734 -6,99 <,0001 
(Blockades B-2,32)*(Blockades B-2,32) 0,3935432 0,035732 11,01 <,0001 


(Latency (real)-33,0154)*(Latency (real)-33,0154)  -0,003664 0,000343 -10,67 ~=,0001 


Figure 34. Reduced regression model. 


The interaction term (Blockades A*Latency) can be interpreted as both factors 
driving the number of hits up additionally when they are at high settings jointly, and vice 


versa. The results show again that latency is the most important controllable factor. 


The model reduction eliminates the communication factors reliability, capacity, 
update frequency, and the sensor range. Their impacts on the outcome are marginal and 
can be neglected in comparison to latency and the red blockades. The scenario 
implemented in MANA is a static one. The changes only occur at 3 points in time. Ina 
more realistic scenario modeling more dynamic elements, the appearance of other 
communication factors in a regression model is possible. The same argument will be 
used in the next section, where thresholds for the communication factors are developed 
by using a regression tree. 

5. Regression Tree for Communication Factors 

This approach identifies thresholds for the communication factors. A regression 


tree is constructed by partitioning exclusively on these factors (see Figure 35). 
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(tition for MeanHITS) 





All Rows 
Count 
Mean 
Std Dev 


1625 
28,548831 
18,043102 




















Count 
Mean 





Latency (real)<28 
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Std Dev 16,735957 
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Pause (real)>=22 
Count 25 
Mean 25,3588 
StdDev 15,40158 














SENSOR Range>=389 
‘Count 100 
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StdDev 15,081537 
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SENSOR Range<389 
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Std Dev 15,477713 
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Latency (real)>=11 
Count 500 
Mean 2689028 
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incompleted tree structure 
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26,648756 
StdDev 16,99033 
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Count 150 
Mean 25,7758 
Std Dev 16,589756 
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Latency (real)<16 
Count 25 
Mean 
Std Dey 15,838763 


24,6988 








Latency (real)>=16 
Count 125 
Mean 25,9912 
Std Dey 16,789059 








Figure 35. 





Count 50 
Mean 23,064 
Std Dev 18,580226 


SENSOR GRID Reliability<24 | 








Count 300 
Mean 27 085233 
Std Dev 17,197862 





SENSOR GRID Capacity<406 














Count 
Mean 21 





Pause (real)<16 


16 578571 
Std Dev 16,888993 
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Mean 
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Pause (real)>=16 


125 
2779456 
17665104 


























SENSOR GRID Capacity>=270 
Count 125 
Mean 26,35264 
StdDev  16,93379 





SENSOR GRID Capacity<270 
Count 50 
Mean 27,1434 
Std Dev 16,934261 











SENSOR Range<557 
Count 7s 
Mean 27 442267 
Std Dev 17,803012 


SENSOR Range>=557 
Count 50 
Mean 28,323 
Std Dev 17,622786 











tree structure is shown. 


Regression tree for the communication factors. Only the left part of the 


The discovery tour through the binary tree starts at the root with all 1625 


observations. In this case, only the left part with “Latency smaller 28 seconds” is shown 


for readability. The latency can also be expressed in a length measure which underlines 


the importance. The convoy runs at 30 km/h through the street channels. In 28 seconds 


it covers about 225 meters, roughly 8 m/s. Following the left branch, the next latency 


split occurs at 11 sec, or about 90 meters. When the amount of time between updates is 


smaller than 22 seconds (185 m) an improvement is noticeable. Sensor range, larger than 


389 m, has also a positive influence on the outcome. 


This chain of thresholds is 


reasonable. It is noted again that the scenario is static. In a dynamic scenario these 


thresholds are likely to drop significantly. 


a7 





Following the branch, “Latency > 11”, it seems that a lack of real-time support 
can be compensated somewhat by a higher reliability and capacity. 

6. Discussion and Conclusions 

The static character of the model and the weakness resulting from the verification 
experiment were stated repeatedly. As such, the findings of this experiment have to be 


read as trends rather than as specific recommendations. 


The analysis shows that the scenario is driven primarily by the enemy behavior. 
The blue force’s reaction relies heavily on precise and non-delayed information. Besides 
the noise factors, latency has the largest impact on the outcome. The other 
communication factors do not have the same importance. But, the model does not allow 
the conclusion that they are unimportant. The lack of a dynamic threat in the scenario 


and the MANA limitations make it impossible to strictly exclude these factors. 
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V. FINAL DISCUSSION AND RECOMMENDATIONS 


The previous chapter has revealed that MANA can produce useful insights. But, 
the author is not totally convinced of their meaningfulness. In particular, the analysis on 
the model shows that it has limitations. However, it is still statistically stable on average, 
as observed by the paired t-test. Due to these limitations, the objective of exploring 
communication aspects cannot be completely and satisfactorily answered. Therefore, the 
author discusses ideas for further research based on the perspectives gained here. These 
studies are difficult when it comes to finding solutions to combine a complex terrain and 
a creative movement algorithm. In this chapter, improvements and _ alternative 
approaches are introduced and discussed. First, suggestions for improving the existing 
model that the author could not implement due to time constraints are discussed. Then, a 
short general excursion is made into network analysis in order to show another potential 
decision making tool. The last discussion point includes a totally different approach. 
Instead of using a time step agent-based model, the potential of a discrete event 


simulation is explained. 


A. IMPROVEMENTS WITHIN THE EXISTING MODEL 

The replication of runs uncovered that MANA’s movement algorithm and an 
urban terrain are a challenging combination. The author chose to let the convoy use the 
whole width of the streets. This led to a situation where the convoy often got stuck and 
chose to look for cover against enemy fire rather than escape. As a result, the MOEs 
were affected negatively. There are two ways to overcome this bad situation. One 
solution is to increase the convoy’s desire to stay away from walls at a specified distance. 
This could keep the convoy in the middle of the road. The alternative, preferred by the 
author, would be to restrict the convoy moves to a line. Imagine a railroad track, one 
narrow guiding line in the middle of the street that allows the convoy’s movements. The 
street width has to be kept large to allow exposure to hostile fire. Of course, the desire 


settings have to be adjusted. The repelling character of blockade or ‘logic helpers’ would 


59 


be increased. With these changes, the convoy has no chance to circumvent a logic helper 


and can not ignore its warning signal. 


B. ALTERNATIVE DECISION TOOL 

The MANA agents move because of their desires. They want to go towards a 
goal or stay away from enemies. A penalty function delivers the movement direction by 
taking the minimum penalty associated with each possible choice. All of these decisions 
are made locally at each time step. A projection into future steps is not implemented, 


which is contrary to real world decision making. 


The modeled scenario is a classic routing problem within a network context. The 
network consists of nodes and arcs. The nodes are crossroads and the arcs are connecting 
streets between two nodes. A blockade is realized as a high are cost. Network analysis 
provides several algorithms?’ to find the best path through such a network. In this case, 
the algorithm would deliver the safest way—given the information available. The 
algorithm results in an ordered list of nodes the convoy has to visit. The nodes can be 
interpreted as waypoints which attract the convoy. Only the next waypoint can be seen 
by the convoy. Every time a new blockade shows up, the associated arc cost is increased 
and the list recalculated. Therefore, movements like turning around and going back to a 


waypoint already visited can be implemented. 


The underlying network structure should be implemented as a third map in 
MANA. The convoy’s decision would be made globally, which is more realistic in an 
urban environment when non-local information is available. This map completes the 
situational awareness. IT-SimBw, the simulation software developed in Germany, 


includes a network map. Preliminary results reported by the creators look promising. 


37 Ravindra K. Ahuja, Thomas L. Magnanti, and James B. Orlin, “Network Flows: Theory, Algorithm, 
and Applications,” 1993. 
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Cy DISCRETE EVENT SIMULATION 

The simulation community has a divided opinion on time step models, like 
MANA. Some analysts totally deny the usefulness of time step models and prefer a 
discrete event simulation (DES).** MANA is a time step model, even though events 
occur and the agents change their states. The author will briefly outline how a DES 


model could look like. 


Picking up the network idea from the previous section, the street map can be 
implemented as a graph with nodes and arcs. The blockades are associated as cost on the 
arcs. The extra cost can be seen as additional exposure to hostile fire and be expressed as 
the number of hits. Each node knows which incoming/outgoing arc is a blocked street 
and how many hits a convoy could suffer passing this node. The convoy itself carries 
lists containing the network information, that is, whether the streets are useable, unknown 
or blocked. One list provides the estimated number of hits the convoy would suffer when 
passing the node given its known blockade setting. Another list provides the estimated 
time it would take to traverse an arc. These databases are regularly updated due to 
changes in the convoy’s situational awareness. The visited nodes are stored in a stack. 
The blockades and their appearance time can be totally randomized, contrary to the 
scripting of MANA. In some sense, blockade setting can be modeled as a smarter way of 


trying to set up ambushes. 


The communication link between a sensor and the convoy can be modeled. For 
example, latency could be achieved by entering an additional delay before the lists 
carried by the convoy are updated. The reliability could be implemented by the decision 
of whether the convoy’s lists are renewed or not. An update frequency is modeled by 


periodically scheduling the next sensor updates. 


In an event driven simulation, the convoy makes a global decision, using a 
shortest path algorithm, regarding which route to take. The arrival at the next node is 


scheduled depending on the arc length (traveling time). The algorithm tries to minimize 


38 Wikipedia.org (November 2006), Definition: “In discrete event simulation, the operation of a 
system is represented as a chronological sequence of events. Each event occurs at an instant in time and 
marks a change of state in the system.” 
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the number of hits while getting to the goal in a reasonable time. The convoy listens to 
the sensor. Every time its graph is updated a new path decision is made—perhaps after 
some delay. In the case where the convoy returns to a prior node an additional delay is 
activated, perhaps resulting in a longer exposure to fire. On its way through the streets 
the convoy collects the number of hits stored at the nodes, or incurs extra costs by 


fighting through a blockade. 


In retrospect, a discrete event simulation on an arc-node structure seems to be 
most appropriate approach to gain insights about communication aspects of convoys in 


urban terrain. 
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APPENDIX 


This Appendix contains tables and figures that complete the experimental design 
and the following analysis. The complete data table for the noise factors combining a 
NOLH design and 8 additional design points is presented. For the analysis part, the Ruby 
code is inserted. In the main text, only one design point was explicitly examined in order 
to assess the model’s quality. The complete analysis on all design points is summarized 


in this Appendix. 


Blockade 
Super_agent 1 | Super_agent 2 | Super_agent3 | Blockade A| Blockade B| Blockade C total 


Low Level 13 








High Level 17 
Decimals 0 
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Table 4. (Appendix) Complete spreadsheet for the noise factors. Design points (DP) 1 
to 17 are the NOLH design points. 18 to 25 are additional ones. In the model 
the super-agents are varied. The resulting blockade settings are calculated in 
the right part. 
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run = 0 


outputFile =File.new("c:/thesis/sep27/HitsMissingDP27sep.txt","w") 
outputFile<<"RUN,EXCURSION,#HITS\n" 
while run < 1 


excursion = 0 
while excursion < 100 
filename = "agt end."+run.to_st+"."+excursion.to_st+".csv" 
path = "c:/thesis/sep27/missingdp/"+filename 
readFile = File.open(path,"r") 
count = 0 
while count < 7 
line = readFile.gets 
values = line.chomp.split(/,/) 
count = count + 1 
end 
hits = values[ 10] 
xpos=values[4] 
ypos=values[5] 


outputFile<<run.to_s+","+excursion.to_st","+hits.to_s+","+xpos.to_s+"- 
"+ypos.to_s+"\n" 

excursion = excursion + | 

end 





Figure 36. (Appendix) Ruby-code used for extracting the important data element out 


of the provide agent-end data file. MANA writes ordered output files. 
These files can be simply accessed, as the variable filename demonstrates. 
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Distributions | 
| Hit Difference | 
Quantiles Moments || Fitted Normal || Test Mean=value | 
100.0% maximum 24,00 Mean 0,76 Parameter Estimates | Hypothesized Value it 
rr ye mi ae Ace Type Parameter Estimate Lower 95% Upper 95% im Estimate — 
pee Fe seks tyeaazas vocation Mu 0,760000 0,056586 1.463434 aise 
3 : De. oh eP ee eae Dispersion Sigma 3545149 3112685 4.118312 sa ‘ 
75.0% quartile 0,00 lower 95% Mean 0,0565655 | ———————__________ tTest 
50.0% median 0,00 N 109 _Goodness-of-Fit Test | Test Statistic 2.1438 
25.0% quartile 0,00 Shapiro-Vvilk VV Test Prob > {t] 0,0345 
10.0% 0,00 W Probe Prob >t 0,0173 
M5 -10 -5 GO 5 10 15 20 25| 25% 0,00 0,362865 0,000 Prob <t 0.9827 
05% 12,00 
0.0% = minimum -12,00 
_ ——Normal(0,76,3,54515) is _ = 
Blockade A=0, Blockade B=1, Blockade C=0 | 
Distributions | 
| Hit Difference | 
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Figure 37. 
quality. Due to readability the 25 design points ar 
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e split. 


(Appendix) The sequence of statistical tests conducted to assess the model 
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10 0 10 20 30 40 50 60 70 80] 25% “3,47 0694505 = <,0001 Prob <t 41,0000 
0.5% -8,00 
0.0% minimum -8,00 
7 —— Normal(6,82,11 41 36) a 
| Blockade A=1, Blockade B=1, Blockade C=0 | 
| Distributions j 
| Hit Difference J 
Quantiles || Moments || Fitted Normal || Test Mean=value 
100.0% maximum 99,00 Mean 611 | Parameter Estimates 7 | Hypothesized Value Q 
eae mes te zinc heed Type Parameter Estimate Lower 95% Upper 95% a Estimate su 
90.0% 21.80 ies oisesos — Lecetion Mu i uldret mmm Sada ame Te 15,3655 
; : email lds! Dispersion Sigma 1536551 1349102 17,84973 si ’ 
75.0% quartile 11,50 lower 95% Mean 3,0611495 t Test 
50.0% median 1,50 N 100 | Goodness-of-FitTest Test Statistic 3.9764 
25.0% quartile 0,00 Shapiro-Vvilk WV Test Prob = ft] 00001 
10.0% -5,90 W Probe Prob >t =,0001 
40 -20 0 20 40 60 80 100] 25% -17,80 0.762453 = <,0001 Prob <t 0.9999 
0.5% -26,00 
0.0% = minimum — -26,00 
— Normal(6,11 15,3655) 
| Blockade A=1, Blockade B=1, Blockade C=1 | 
| Distributions J 
| Hit Difference | 
_ Quantiles || Moments || Fitted Normal | Test Mean=value 
100.0% maximum 102,0 Mean 8,29 | Parameter Estimates Hypothesized Value it} 
eas yes ag path picid Type Parameter Estimate Lower 95% Upper 95% ae Eamets, a 
90.0% 257 Apes 42,084607 eh ey pase nee eel Std D 18,9728 
: ’ eo ere donee: Dispersion Sigma 1897276 1665821 22,04018 " 
75.0% quartile 14,0 lower 95% Mean 4,5253931  —___ = t Test 
50.0% — median 50 N 100 | Goodness-of-FitTest | Test Statistic 4.3694 
25.0% quartile oo Shapiro-Wvilk VV Test Prob = ft] <,0001 
10.0% 50 W Probe Prob >t <,0001 
60 -40 -20 0 20 40 60 80 100120] 25% -32,0 0.833173 = <,0001 Prob <t 41,0000 
0.5% -45,0 
0.0% minimum -45,0 
— Normal(8,29,18,9728) 
Blockade A=1, Blockade B=2, Blockade C=4 
Distributions 
Hit Difference 
Quantiles | Moments | Fitted Normal || Test Mean=value 
100.0% maximum 161.0 Mean 8,39 [ Parameter Estimates | Hypothesized Value oO 
ae ean = ae — Type Parameter Estimate Lower 95% Upper 95% ies Estimate is 
ae aaa ae teoieder vocation Mu 839000 (0,76350 16,01650 Sises 
; x it «Upper. secede Dispersion Sigma 3843580 33,74690 44,6492 Bi ‘ 
75.0% quartile 21,0 lower 95% Mean 0,7635027 9 —————__________ t Test 
50.0% median 40 oN 109 | Goodness-of-Fit Test Test Statistic 2.1829 
25.0% quartile -3,0 Shapiro-Wvilk VV Test Prob = ttl 0,0314 
10.0% -24,0 We Probe Prob >t 0,0157 
25% -80,4 0842549 = <,0001 Prob <t 0,9843 
0.5% -116,0 
0.0% minimum = -116,0 





—— Normal(8,39,38,4358) 


Figure 38. 


66 


(Appendix) The sequence of statistical tests conducted to assess the model 
quality. Due to readability the 25 design points are split. 





| Blockade A=1, Blockade B=3, Blockade C=1 








| Distributions 






































































































































| Hit Difference 
| Quantiles || Moments || Fitted Normal | Test Mean=value i 
100.0% maximum 116,0 Mean 314 | Parameter Estimates | Hypothesized Value it) 
ane pee ie a a pe Type Parameter Estimate Lower 95% Upper 95%. Pa esting? pe 
aie ae a A wayaeqn —recstion Mu 914000484058 13,7382 sig 
: : eee seul p Dispersion Sigma 2318002  20,35222 26,92766 e: ' 
75.0% quartile 22,0 lower 95% Mean 45405802, ~ t Test 
50.0% median = 4.0N 100 | Goodness-of-Fit Test | Test Statistic 3,9431 
25.0% quartile -5,0 Shapiro-Wvilk VV Test Prob = ttl 0,002 
10.0% 128 WW Probew Prob >t <,0001 
-40 -20 0 20 40 60 80 100 120] 25% -29,5 0e94650 «=< 0001 Prob <t 0,9999 
05% -32,0 
0.0% minimum -32,0 
— Normal(9,14,23,18) 
| Blockade A=1, Blockade B=4, Blockade C=3 | 
| Distributions | 
| Hit Difference a _ 7 = a} 
| Quantiles | Moments || Fitted Normal | Test Mean=value | 
.: - —f Ht oes 100.0% maximum  181,0 Mean 11,37 | Parameter Estimates | Hypothesized Value 9 
— ae Ha = ae pean Type Parameter Estimate Lower 95% Upper 95%. im Estinete "1 a 
aoe es vais tacoes7s — Loetion Mu 11,3700 2.93082 1980988 pie 
: ; if Upper el Dad Dispersion Sigma 4253404 3734518 49,41074 cl ‘ 
75.0% quartile 24,0 lower 95% Mean 2,9303237 — ————______________ t Test 
50.0% — median 40 N 100 | Goodness-of-Fit Test _ Test Statistic 2,6732 
25.0% quartile 88 Shapiro-Wvilk VV Test Prob = {tl 0,0088 
10.0% 19,9 W Probe Prob >t 0,0044 
150-100 -50 0 50 100 150 200] 25% -80,3 ogs7186 = <,0001 Prob <t 0.9956 
05% -146,0 
0.0% minimum = -146,0 
— Normal(11 37 42,534) 
| Blockade A=2, Blockade B=1, Blockade C=3 | 
| Distributions | 
| HitDifference 2 Ea 7-22 ae 
| Quantiles | Moments || Fitted Normal | Test Mean=value 
100.0% maximum 159.0 Mean 10,36 Parameter Estimates | Hypothesized Value 0 
eae this = a pene Type Parameter Estimate Lower 95% Upper 95% a Estimate, eee 
ate Ga Kia ‘oisqioa Locsin Mu 10,36000«4,98589 1615411 aati 
: : Pm ate eae Dispersion Sigma 29,2098 © -25,638686 —33,92208 ey : 
75.0% quartile 22,0 lower 95% Mean 4,5658918 4 tTest 
50.0% median 50 ON 109 | Goodness-of-Fit Test | Test Statistic 3.5478 
25.0% quartile As Shapiro-Wvilk WV Test Prob = ttl 0,0006 
10.0% 165 W Probe Prob >t 0,0003 
25% -46,4 0883189 © <,0001 Prob <t 0,9997 
05% -70,0 
0.0% — minimum = —--70,0 
— Normal10,36,29,201) 
| Blockade A=2, Blockade B=3, Blockade C=2 
| Distributions 
| Hit Difference 
Quantiles | Moments || Fitted Normal | Test Mean=value 
100.0% maximum 144.0 Mean 12,97 | Parameter Estimates Hypothesized Value it} 
- eg ‘ = : = ei pee zi Type Parameter Estimate Lower 95% Upper 95% oe Estimate i ge 
oA ae shear Noassos rection Mu 1297000 6.08494 1985508 ore 
: ; ec em ets! Dispersion Sigma 3469913  30,46607 40,30912 ey ! 
75.0% quartile 19,5 lower 95% Mean 6,0849405 © ———______________ tTest 
50.0% median 55 N 109 _ Goodness-of-Fit Test Test Statistic 3,7378 
25.0% quartile -3,8 Shapiro-Wvilk VV Test Prob > {tl 0,0003 
10.0% 10,9 WW Probew Prob >t 0.0002 
25% 58,5 0840938 = -<,0001 Prob <t 0.9998 
05% -68,0 
0.0% — minimum —-68,0 


—— Normal(12,97,34,6991) 


Figure 39, 
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(Appendix) The sequence of statistical tests conducted to assess the model 
quality. Due to readability the 25 design points are split. 





| Blockade A=2, Blockade B=4, Blockade C=1 









































































































































| Distributions Sas J 
| Hit Difference } 
Quantiles || Moments || Fitted Normal | Test Mean=value } 
se — TPE — + + |P 100.0% maximum —165,0 Mean 17,32 | Parameter Estimates | Hypothesized Value 0 
—- rs ss ie = oath peste Type Parameter Estimate Lower 95% Upper 95% a ciniae oe 
lag as SBCA seaosera Locsin Mu 17,32000-8,74833 2589387 ‘505i 
: : ity URS eukecw: Dispersion Sigma 43,20935 © 37,93811 _50,19523 ey. ' 
75.0% quartile 31,0 lower 95% Mean 6,74632720© ———______________ t Test 
50.0% — median 100 N 109 | Goodness-of-Fit Test Test Statistic 4,084 
25.0% quartile -3,8 Shapiro-Vvilk V¥ Test Prob > tl 0,0001 
10.0% -358 WW Probev Prob >t <,0001 
50 0 50 100 150 25% -59,4 0.928616 —_<,0001 Prob <t 0,9999 
05% 10 
0.0% — minimum —--74,0 
—Normal(17,32,43,2094) 
| Blockade A=2, Blockade B=5, Blockade C=4 J 
| Distributions 
| Hit Difference } 
Quantiles | Moments || Fitted Normal || Test Mean=value | 
100.0% maximum  180,0 Mean 1579 | Parameter Estimates : | Hypothesized Value Q 
ae pe =e oak pase Type Parameter Estimate Lower 95% Upper 95% 3 ibis en 
we. ay pee) casore1g Location Mu 1879000 7.27238 2430762 jo sons 
! : maaan eee eet Dispersion Sigma 4292683  37,69005 49,88704 ee ’ 
75.0% quartile 22,8 lower 95% Mean 72723849 = t Test 
50.0% median 35 N 100 Goodness-of-Fit Test Test Statistic 3.6784 
25.0% quartile -9,0 Shapiro-Vvilk VW Test Prob = {t| 0,0004 
10.0% 169 WW Probev Prob >t 0,0002 
100 150 200] 25% 585 0974512 = <,0001 Prob <t 0,9998 
0.5% -82,0 
0.0% minimum -82,0 
——Normal(15,79,42,9268) 
| Blockade A=3, Blockade B=1, Blockade C=2 
| Distributions 
| Hit Difference } 
Quantiles ___|| Moments || Fitted Normal || Test Mean=value } 
100.0% maximum 99,00 Mean 12,11 | Parameter Estimates | Hypothesized Value a 
= = ee 2 ae pple eee Type Parameter Estimate Lower 95% Upper 95% oo Saw as 
ere ste sntell teopiatg Location Mu 1211000 6.98868 17,28132 Paros 
; ; foe. eee ioe Dispersion Sigma 2596147 22,79435 + 30,15880 ste ' 
75.0% quartile 23,75 lower 95% Mean 69586813 | _——______________ t Test 
50.0% median 6,50 N 100 | Goodness-of-Fit Test Test Statistic 4,6646 
25.0% quartile 1,75 Shapiro-Vvilk VV Test Prob > {tl =,0001 
10.0% -9,80 WW Probev Prob >t <,0001 
60 -40 -20 0 20 40 60 80 100] 25% -39,38 0878005 = _<,0001 Prob <t 41,0000 
05% -43,00 
0.0% minimum  -43,00 
——Normal(12,11,25,9615) 
Blockade A=3, Blockade B=2, Blockade C=5 
| Distributions 
Hit Difference 
Quantiles | Moments | Fitted Normal || Test Mean=value ; 
100.0% maximum  294,0 Mean 13,81 | Parameter Estimates | Hypothesized Value 0 
he re = ae mee ee Type Parameter Estimate Lower 95% Upper 95% as Esmee Bae 
sae 7 aay sparovaa — cecation Mu 131000 424727 23,7273 rae 
i ; i SUR DEE: alee Dispersion Sigma 4819399 4231485 55,98576 ad " 
75.0% quartile «26,8 lower 95% Mean 42472672. t Test 
50.0% median 60 N 100 | Goodness-of-Fit Test Test Statistic 2,8655 
25.0% = quartile 58 Shapiro-Vvilk W Test Prob = ft] 0.0051 
10.0% 29 We Probev Prob >t 0,025 
M50 -80 0 50 100150 200 250 300] 25% -66,2 o7740568 = <,0001 Prob =t 0,9975 
05% 1290 
0.0% minimum — -129,0 


— Normal(13,81 48,194) 


Figure 40. 
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(Appendix) The sequence of statistical tests conducted to assess the model 
quality. Due to readability the 25 design points are split. 





| Blockade A=3, Blockade B=3, Blockade C=3 








| Distributions 




















































































































| Hit Difference 
Quantiles || Moments | Fitted Normal || Test Mean=value 
100.0% maximum 203,0 Mean 15,68 | Parameter Estimates | Hypothesized Value 0 
7 = nee = re eoed al Type Parameter Estimate Lower 95% Upper 95% Pius Estimate ge 
hy aS sitet cetorias Location Mu 1568000 8.23286 2312714 St 
: ‘ 1G gee cones) Dispersion Sigma 3753191  32,95327 43,59989 ee, ‘ 
75.0% quartile 25,3 lower 95% Mean 8,2328554 | _————______________ t Test 
50.0% median 80 N 100 | Goodness-of-Fit Test | Test Statistic 4.1778 
25.0% quartile 48 Shapiro-Wvilk VV Test Prob > ft} =,0001 
10.0% 146 We Probevv Prob >t <,0001 
50 100 150 200 25% -55,0 08376833 © <,0001 Prob <t 41,0000 
05% -76,0 
0.0% — minimum —--76,0 
—Normal(15,68,37,5319) a 
| Blockade A=3, Blockade B=4, Blockade C=4 
| Distributions 
| Hit Difference =sineae aS == == : 
Quantiles || Moments | Fitted Normal || Test Mean=value 
100.0% maximum 208,0 Mean 15,28 | Parameter Estimates | Hypothesized Value it} 
— per = nade roland Type Parameter Estimate Lower 95% Upper 95% oe Estimate 
piles ee tae seosdaga ~—recation Mu 15,2800 530565 25,25434 Pee 
; ; Me PRE ean ee Dispersion Sigma 50,28839 4413599 §8,39554 sd , 
750% querile 37,5 lower 85%Mean §,3058615. tTest 
50.0% median 11,5 N 100 | Goodness-of-FitTest | Test Statistic 3,0397 
25.0% quartile -60 Shapiro-Wvilk VV Test Prob > {t} 0,0030 
10.0% -28,0 W Probew Prob >t 0,0015 
200 -100-50 0 50 100150200 250] 2.5% 1138 016378 =< 0001 Prob <t 0.9985 
05% -185,0 
0.0% minimum = -185,0 
— Normal(15,28,50,2684) 
| Blockade A=4, Blockade B=1, Blockade C=2 al 
| Distributions 
| Hit Difference 
Quantiles || Moments | Fitted Normal || Test Mean=value 
100.0% maximum 236,0 Mean 47 57 | Parameter Estimates | Hypothesized Value 0 
Bae prea z ae peel ph Type Parameter Estimate Lower 95% Upper 95%. as Estimate ed 
ah a5 ora shoranss — Locetion Mu 1757000 8.88894 26,27908 Perry 
: . re Upper cana Dispersion Sigma 4386141 3851062 50,9272 ev i 
75.0% quartile 27,8 lower 95% Mean 88669445 t Test 
50.0% median 105 N 100 | Goodness-of-Fit Test Test Statistic 4,058 
25.0% quartile -3,0 Shapiro-Wvilk V¥ Test Prob = {tl 0.0001 
10.0% 15,0 WW Probev Prob >t <,0001 
150-100 -80 O S50 100 150 200 250] 25% -78, 0e40787 =< 0001 Prob <t 0,9999 
05% 1140 
0.0% minimum =-114,0 
— Normal(17,57,43,8614) 
Blockade A=4, Blockade B=2, Blockade C=4 
Distributions 
Hit Difference 
Quantiles | Moments || Fitted Normal | Test Mean=value 
100.0% maximum 112.0 Mean 11,93 f Parameter Estimates Hypothesized Value 0 
apy ibs : ey oem eae Type Parameter Estimate Lower 95% Upper 95% ag Estimate aa bes 
ne Be ee ii taoaveor —cecation Mu 11,9300 482207 19,23793 eee 
: ' ee URES: con ease Dispersion Sigma 3683028 32,33724 -42,78483 Si i 
75.0% quartile 26,8 lower 95% Mean 46220727 © ——__________ t Test 
50.0% = median 7Oo N 100 Goodness-of-Fit Test Test Statistic 32392 
25.0% — quartile -58 Shapiro-Vvilk VW Test Prob = ft] 0,0016 
10.0% -23,6 W Probe Prob >t 0,0008 
1150 -100 -50 O 50 100 25% NA 0934972 = -=,0001 Prob =t 0,992 
0.5% -139,0 
0.0% minimum  -139,0 


—— Normal(11 ,93,36,8303) 


Figure 41. 


quality. Due to readability the 25 design points are split. 
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(Appendix) The sequence of statistical tests conducted to assess the model 





| Blockade A=4, Blockade B=3, Blockade C=5 





| Distributions 








| Hit Difference 


























































































































Quantiles || Moments | Fitted Normal | Test Mean=value 
100.0% maximum  150,0 Mean 13,29 | Parameter Estimates Hypothesized Value a 
99.5% 150,0 Std Dev 45386176 Type Parameter Estimate Lower 95% Upper a5%  ctual Estimate 13,29 
97.5% 126,1 Std Err Mean 45386176 : df 99 
90.0% 746 95% Mi 22,295602 Eocation |. Ma 122000 Std D 45,3862 
; : pam ied cones Dispersion Sigma 45,38618 Ad ’ 
75.0% quartile 39,5 lower 95% Mean 4,2643981) © -————______ tTest 
§0.0% median 50 N 100 © Goodness-of-Fit Test | Test Statistic 2,9282 
25.0% quartile 10,0 Shapiro-Vvilk VW Test Prob > ttl 0,0042 
10.0% -32,5 W Probew Prob >t 0,0021 
200-150-100 -50 0 50 100150 200] 25% -95,9 0.939138 —-0,0002 Prob <t 09979 
05% -159,0 
0.0% minimum = -159,0 
a Norinai(t3,29,45,3062) ___ a 
Blockade A=4, Blockade B=5, Blockade C=3 
| Distributions 
Difference 
Quantiles || Moments || Fitted Normal || Test Mean=value 
100.0% maximum 230,0 Mean 32,51 [ Parameter Estimates | Hypothesized Value ii} 
99.5% 230,0 Std Dev 58,096488 Type Parameter Estimate Lower 95% Upper 95% Actual Estimate 32,51 
97 5% 198,9 Std Err Mean 58096488 : df 99 
90.0% 113,8 upper 95% Mean 44,037604 Eocetion Mu 5281009 Std Dey 58,0965 
‘ 7 i ‘ Dispersion Sigma 58,09649 
75.0% quartile 57,0 lower 95% Mean = 20,982396 tTest 
50.0% median 17,0 N 100 | Goodness-of-Fit Test | Test Statistic §,5959 
25.0% quartile 58 Shapiro-Wvilk VW Test Prob > It] <,0001 
10.0% -24,9 W Probe Prob >t <,0001 
Oo 50 100 150 200 250] 25% -50,8 0.902161 <,0001 Prob <t 14,0000 
0.5% -67,0 
0.0% minimum -67,0 
— Normal¢32,51 58,0965) - 
| Blockade A=5, Blockade B=2, Blockade C=2 
| Distributions 
| Hit Difference 
Quantiles ___ || Moments: | Fitted Normal __ || Test Mean=value 
100.0% maximum 232,0 Mean 14,83 Parameter Estimates Hypothesized Value it) 
99.5% 232,0 Std Dev 38427435 Type Parameter Estimate Lower 95% Upper a5%  ctual Estimate 14,83 
97.5% 93,4 Std Err Mean 3,8427435 " df 99 
90.0% 50,8 95% Mi 22,454837 Eocene eam Std D 38,4274 
2 2 1 haces rail z Dispersion Sigma 3842744 sat : 
75.0% quartile 28,8 lower 95% Mean 7,2051632.©© ——————________~ t Test 
50.0% median 75 N 109 _Goodness-of-Fit Test | Test Statistic  3,8592 
| 25.0% quartile -40 Shapiro-Wvilk V¥ Test Prob = tt| 0.0002 
P L. 10.0% 118 W Probe Prob >t 0,0001 
150-100 -50 0 50 100 150 200 250) 2.5% -59,2 0,805772 <,0001 Prob <t 0,9999 
05% -136,0 
0.0% minimum = -136,0 
—— Normal(1 483,38 ,4274) 
| Blockade A=5, Blockade B=4, Blockade C=1 
| Distributions 
| Hit Difference 5 
Quantiles || Moments || Fitted Normal | Test Mean=value 
100.0% maximum 205,0 Mean 22,56 [ Parameter Estimates Hypothesized Value it) 
99.5% 205,0 Std Dev 44953246 Type Parameter Estimate Lower 95% Upper a5% tual Estimate 22,56 
97.5% 1621 Std Err Mean 44953246 Locati Mi e000 df 99 
90.0% 77,9 upper 95% Mean 31,479699 Seana: 229000 Std Dev 44,9532 
‘i . : : Dispersion Sigma 44,95325 " 
75.0% quartile 35,0 lower 95% Mean 13640301 ———____________. t Test 
50.0% median 12,0 N 100 | Goodness-of-Fit Test | Test Statistic $0185 
25.0% quartile -2,0 Shapiro-Wik W Test Prob > ft <,0001 
10.0% -3,0 W Probevy Prob >t <,0001 
50 100 150 200 2.5% -52,9 0,835160 <,0001 Prob <t 41,0000 
05% -66,0 
0.0% minimum -66,0 


—— Normal(22,56,44,9532) 


Figure 42. 
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(Appendix) The sequence of statistical tests conducted to assess the model 
quality. Due to readability the 25 design points are split. 


Blockade A=5, Blockade B=5, Blockade C=5 


























Distributions 
Hit Difference 
Quantiles Moments || Fitted Normal Test Mean=value 
100.0% maximum 230,0 Mean 39,08 Parameter Estimates Hypothesized Value it} 
a rea = Eus pet 7 Type Parameter Estimate Lower 95% Upper 95% a coum ee 
ane. ja mn ae dogsoe1g vocation Mu 3908000 2880779 4985221 ae 
: ; bodies ae Dispersion Sigma 5328154 46,78155 —61,89585 sf ' 

75.0% quartile 68,8 lower 95% Mean 28,507786 — t Test 
50.0% median 315 N 100 | Goodness-of-Fit Test Test Statistic 7.3346 
25.0% quartile 43 Shapiro-VWvilk WV Test Prob = ft] «,0001 
10.0% -20,7 W Probe Prob >t <,0001 

}100 -50 0 50 100 150 200 250] 25% -44,9 0958710 = 0,0033 Prob <t 41,0000 
05% -92,0 
0.0% minimum -92,0 


—— Normal(39,08,53,2815) 


Figure 43. 


(Appendix) The sequence of statistical tests conducted to assess the model 


quality. Due to readability the 25 design points are split. 
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